PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device

On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a third
party package. It is what the builtin logical replication is derived
from, but it is not the same thing. So what version of pg_logical are
you using?

ERROR:  could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who faced this problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

The error says you do.
Where is pg_logical/snapshots/ mounted?
Are there specific restrictions on that mount?

Best Regard

--
Jaurès FOUTE

--
Adrian Klaver
adrian.klaver@aklaver.com

On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:

On 6/30/20 11:03 AM, FOUTE K. Jaur�s wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a third party
package. It is what the builtin logical replication is derived from, but it
is not the same thing. So what version of pg_logical are you using?

ERROR: �could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who�faced this�problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

The error says you do.
Where is pg_logical/snapshots/ mounted?
Are there specific restrictions on that mount?

I would also look at your kernel log.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

Le mar. 30 juin 2020 à 21:23, Bruce Momjian <bruce@momjian.us> a écrit :

On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:

On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a third party
package. It is what the builtin logical replication is derived from, but

it

is not the same thing. So what version of pg_logical are you using?

I only install Postgresql 12. I dont use pg_logical

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who faced this problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

The error says you do.
Where is pg_logical/snapshots/ mounted?

/var/lib/postgresql/12/main/pg_logical (default location)

Are there specific restrictions on that mount?

[image: image.png]

I would also look at your kernel log.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

--
Jaurès FOUTE

Le mar. 30 juin 2020 à 21:23, Bruce Momjian <bruce@momjian.us> a écrit :

On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:

On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a third party
package. It is what the builtin logical replication is derived from, but

it

is not the same thing. So what version of pg_logical are you using?

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who faced this problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

The error says you do.
Where is pg_logical/snapshots/ mounted?
Are there specific restrictions on that mount?

I would also look at your kernel log.

[image: image.png]

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: *jauresfoute@gmail.com <jauresfoute@gmail.com>*,
jauresmelkiore@yahoo.fr
Compte Skype: jauresmelkiore

On Tue, Jun 30, 2020 at 10:03:52PM +0100, FOUTE K. Jaur�s wrote:

Le�mar. 30 juin 2020 �21:23, Bruce Momjian <bruce@momjian.us> a �crit�:

On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:

On 6/30/20 11:03 AM, FOUTE K. Jaur�s wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a third party
package. It is what the builtin logical replication is derived from, but

it

is not the same thing. So what version of pg_logical are you using?

ERROR: �could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who�faced this�problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

The error says you do.
Where is pg_logical/snapshots/ mounted?
Are there specific restrictions on that mount?

I would also look at your kernel log.

image.png

OK, are you using SAN/iSCSI or NAS, which uses Ethernet to communicate
with the storage?

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

On 6/30/20 1:47 PM, FOUTE K. Jaurès wrote:

Le mar. 30 juin 2020 à 21:23, Bruce Momjian <bruce@momjian.us
<mailto:bruce@momjian.us>> a écrit :

On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:

On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on

Ubuntu 16.04

and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a

third party

package. It is what the builtin logical replication is derived

from, but it

is not the same thing. So what version of pg_logical are you using?

I only install Postgresql 12. I dont use pg_logical

My mistake, I remembered the location for the logical files wrong.

/var/lib/postgresql/12/main/pg_logical (default location)

From /var/lib/postgresql/12/main/pg_logical what does du -sh show?
Please show results as text, you can copy from terminal screen.

Are there specific restrictions on that mount?

image.png

I would also look at your kernel log.

--
  Bruce Momjian  <bruce@momjian.us <mailto:bruce@momjian.us>>
https://momjian.us
  EnterpriseDB https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee

--
Jaurès FOUTE

--
Adrian Klaver
adrian.klaver@aklaver.com

FOUTE K. Jaur�s wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04 and
all is working fine until i faced this error below.

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who faced this problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

Best Regard

--
Jaur�s FOUTE

If you really haven't run out of space,
you might have run out of inodes.
The -i option of df should show you.
100,000 empty files could cause that.
I wouldn't think that renaming would
require a new inode, but I also wouldn't
think that renaming would require any
more space on a file system.

cheers,
raf

Le mar. 30 juin 2020 à 22:36, Adrian Klaver <adrian.klaver@aklaver.com> a
écrit :

On 6/30/20 1:47 PM, FOUTE K. Jaurès wrote:

Le mar. 30 juin 2020 à 21:23, Bruce Momjian <bruce@momjian.us
<mailto:bruce@momjian.us>> a écrit :

On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:

On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on

Ubuntu 16.04

and all is working fine until i faced this error below.

From below it looks like you are using pg_logical which is a

third party

package. It is what the builtin logical replication is derived

from, but it

is not the same thing. So what version of pg_logical are you

using?

I only install Postgresql 12. I dont use pg_logical

My mistake, I remembered the location for the logical files wrong.

/var/lib/postgresql/12/main/pg_logical (default location)

From /var/lib/postgresql/12/main/pg_logical what does du -sh show?
Please show results as text, you can copy from terminal screen.

root@hybride:/var/lib/postgresql/12/main/pg_logical# du -sh
16K .

Are there specific restrictions on that mount?

image.png

I would also look at your kernel log.

--
Bruce Momjian <bruce@momjian.us <mailto:bruce@momjian.us>>
https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

--
Jaurès FOUTE

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: *jauresfoute@gmail.com <jauresfoute@gmail.com>*,
jauresmelkiore@yahoo.fr
Compte Skype: jauresmelkiore

Le mer. 1 juil. 2020 à 00:11, raf <raf@raf.org> a écrit :

FOUTE K. Jaurčs wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04

and

all is working fine until i faced this error below.

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who faced this problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

Best Regard

--
Jaurčs FOUTE

If you really haven't run out of space,
you might have run out of inodes.
The -i option of df should show you.
100,000 empty files could cause that.
I wouldn't think that renaming would
require a new inode, but I also wouldn't
think that renaming would require any
more space on a file system.

The result of df -i

Sys. de fichiers Inœuds IUtil. ILibre IUti% Monté sur
udev 3065149 433 3064716 1% /dev
tmpfs 3072780 665 3072115 1% /run
/dev/sdb2 59973632 356029 59617603 1% /
tmpfs 3072780 7 3072773 1% /dev/shm
tmpfs 3072780 10 3072770 1% /run/lock
tmpfs 3072780 17 3072763 1% /sys/fs/cgroup
/dev/sdb1 0 0 0 - /boot/efi
tmpfs 3072780 19 3072761 1% /run/user/108
tmpfs 3072780 5 3072775 1% /run/user/1001
tmpfs 3072780 5 3072775 1% /run/user/1000

cheers,
raf

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: *jauresfoute@gmail.com <jauresfoute@gmail.com>*,
jauresmelkiore@yahoo.fr
Compte Skype: jauresmelkiore

On 6/30/20 10:51 PM, FOUTE K. Jaurès wrote:

 From /var/lib/postgresql/12/main/pg_logical what does du -sh  show?
Please show results as text, you can copy from terminal screen.

root@hybride:/var/lib/postgresql/12/main/pg_logical# du -sh
16K .

Has the problem occurred again?

--
Adrian Klaver
adrian.klaver@aklaver.com

On 2020-Jun-30, FOUTE K. Jaur�s wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04 and
all is working fine until i faced this error below.

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

What are your fsync and wal_sync_method settings?

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Le mer. 1 juil. 2020 à 15:45, Adrian Klaver <adrian.klaver@aklaver.com> a
écrit :

On 6/30/20 10:51 PM, FOUTE K. Jaurès wrote:

From /var/lib/postgresql/12/main/pg_logical what does du -sh show?
Please show results as text, you can copy from terminal screen.

root@hybride:/var/lib/postgresql/12/main/pg_logical# du -sh
16K .

Has the problem occurred again?

Yes
2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive data from
WAL stream: ERROR: could not rename file "pg_logical / snapshots /
1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots / 1E-8567EB0.snap":
No space left on device

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: *jauresfoute@gmail.com <jauresfoute@gmail.com>*,
jauresmelkiore@yahoo.fr
Compte Skype: jauresmelkiore

Le mer. 1 juil. 2020 à 17:11, Alvaro Herrera <alvherre@2ndquadrant.com> a
écrit :

On 2020-Jun-30, FOUTE K. Jaurčs wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04

and

all is working fine until i faced this error below.

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

What are your fsync and wal_sync_method settings?

All the two setting are commented
#fsync = on
#wal_sync_method = fsync

--
Įlvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: *jauresfoute@gmail.com <jauresfoute@gmail.com>*,
jauresmelkiore@yahoo.fr
Compte Skype: jauresmelkiore

On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:

Has the problem occurred again?

Yes
2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive data from
WAL stream: ERROR: could not rename file "pg_logical / snapshots /
1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots /
1E-8567EB0.snap": No space left on device

What are the permissions on?:

pg_logical/snapshots/

--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
Compte Skype: jauresmelkiore

--
Adrian Klaver
adrian.klaver@aklaver.com

On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:

Le mer. 1 juil. 2020 à 15:45, Adrian Klaver <adrian.klaver@aklaver.com
<mailto:adrian.klaver@aklaver.com>> a écrit :

On 6/30/20 10:51 PM, FOUTE K. Jaurès wrote:

       From /var/lib/postgresql/12/main/pg_logical what does du

-sh  show?

     Please show results as text, you can copy from terminal screen.

root@hybride:/var/lib/postgresql/12/main/pg_logical# du -sh
16K .

Has the problem occurred again?

Yes
2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive data from
WAL stream: ERROR: could not rename file "pg_logical / snapshots /
1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots /
1E-8567EB0.snap": No space left on device

Also look in system log for any events around 2020-07-01 19: 19

--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
Compte Skype: jauresmelkiore

--
Adrian Klaver
adrian.klaver@aklaver.com

Le mer. 1 juil. 2020 à 22:15, Adrian Klaver <adrian.klaver@aklaver.com> a
écrit :

On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:

Has the problem occurred again?

Yes
2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive data from
WAL stream: ERROR: could not rename file "pg_logical / snapshots /
1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots /
1E-8567EB0.snap": No space left on device

What are the permissions on?:

pg_logical/snapshots/

postgres

root@hybride:/var/lib/postgresql/12/main/pg_logical# ll
total 20
drwx------ 4 postgres postgres 4096 juil. 1 20:21 ./
drwx------ 19 postgres postgres 4096 juil. 1 20:16 ../
drwx------ 2 postgres postgres 4096 mars 31 11:58 mappings/
-rw------- 1 postgres postgres 56 juil. 1 20:21 replorigin_checkpoint
drwx------ 2 postgres postgres 4096 mars 31 11:58 snapshots/

--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
Compte Skype: jauresmelkiore

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: *jauresfoute@gmail.com <jauresfoute@gmail.com>*,
jauresmelkiore@yahoo.fr
Compte Skype: jauresmelkiore

On 7/1/20 2:21 PM, FOUTE K. Jaurès wrote:

Le mer. 1 juil. 2020 à 22:15, Adrian Klaver <adrian.klaver@aklaver.com
<mailto:adrian.klaver@aklaver.com>> a écrit :

On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:

     Has the problem occurred again?

Yes
2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive

data from

WAL stream: ERROR: could not rename file "pg_logical / snapshots /
1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots /
1E-8567EB0.snap": No space left on device

What are the permissions on?:

pg_logical/snapshots/

postgres
root@hybride:/var/lib/postgresql/12/main/pg_logical# ll
total 20
drwx------  4 postgres postgres 4096 juil.  1 20:21 ./
drwx------ 19 postgres postgres 4096 juil.  1 20:16 ../
drwx------  2 postgres postgres 4096 mars  31 11:58 mappings/
-rw-------  1 postgres postgres   56 juil.  1 20:21 replorigin_checkpoint
drwx------  2 postgres postgres 4096 mars  31 11:58 snapshots/

Well replication is between two instances, so are you sure you are
looking at the correct data directory?

--
Jaurès FOUTE
Technology Consultant
ISNOV SARL - Business Technology Consulting
Tel: +237 79395671 / +237 96248793
Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
Compte Skype: jauresmelkiore

--
Adrian Klaver
adrian.klaver@aklaver.com

FOUTE K. Jaurès wrote:

Le mer. 1 juil. 2020 à 00:11, raf <raf@raf.org> a écrit :

FOUTE K. Jaurčs wrote:

Hi everyone,

I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04

and

all is working fine until i faced this error below.

ERROR: could not rename file
"pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
"pg_logical/snapshots/1A-7C00D890.snap": No space left on device

Is there someone who faced this problem?
Any idea how I can solve it ?

BTW: I don't have any problem with space.

Best Regard

--
Jaurčs FOUTE

If you really haven't run out of space,
you might have run out of inodes.
The -i option of df should show you.
100,000 empty files could cause that.
I wouldn't think that renaming would
require a new inode, but I also wouldn't
think that renaming would require any
more space on a file system.

The result of df -i

Sys. de fichiers Inœuds IUtil. ILibre IUti% Monté sur
udev 3065149 433 3064716 1% /dev
tmpfs 3072780 665 3072115 1% /run
/dev/sdb2 59973632 356029 59617603 1% /
tmpfs 3072780 7 3072773 1% /dev/shm
tmpfs 3072780 10 3072770 1% /run/lock
tmpfs 3072780 17 3072763 1% /sys/fs/cgroup
/dev/sdb1 0 0 0 - /boot/efi
tmpfs 3072780 19 3072761 1% /run/user/108
tmpfs 3072780 5 3072775 1% /run/user/1001
tmpfs 3072780 5 3072775 1% /run/user/1000

So that's not it. It would take ~60 million
files to fill up your / inode table. I can't
think of another explanation for that error
message if df without -i also shows free space.

I tried googling for "No space left on device"
and everything suggested checking inodes with "df -i".
One suggested looking for files that were deleted
but that were still open by a process (can be found
with "lsof / | grep deleted") but if that were the
problem, df would show a full disk (but du wouldn't
be able to account for it).

Could it be some other "device"? like shared memory
segment space or IPC resources? That seems unlikely if
the operation was just a file system rename. Perhaps
strace/stap could help check if it really was a file
system rename that went wrong (if there's any reason to
doubt it). It does seem to clearly be a file rename
though.

If there any chance that the drive is failing? But
you'd expect "I/O error" messages if that were the case.

Is it on a journalling file system and could the journal
be the device that is full? I don't know how to check
that (or if it even makes sense).

cheers,
raf

raf <raf@raf.org> writes:

FOUTE K. Jaurès wrote:

The result of df -i

Sys. de fichiers Inœuds IUtil. ILibre IUti% Monté sur
udev 3065149 433 3064716 1% /dev
tmpfs 3072780 665 3072115 1% /run
/dev/sdb2 59973632 356029 59617603 1% /
tmpfs 3072780 7 3072773 1% /dev/shm
tmpfs 3072780 10 3072770 1% /run/lock
tmpfs 3072780 17 3072763 1% /sys/fs/cgroup
/dev/sdb1 0 0 0 - /boot/efi
tmpfs 3072780 19 3072761 1% /run/user/108
tmpfs 3072780 5 3072775 1% /run/user/1001
tmpfs 3072780 5 3072775 1% /run/user/1000

So that's not it. It would take ~60 million
files to fill up your / inode table. I can't
think of another explanation for that error
message if df without -i also shows free space.

I'm going to take a shot in the dark and ask if the root file system
is XFS. It doesn't take too much googling to find out that XFS has a
reputation for reporting ENOSPC when there seems to be more than enough
room left. Apparently, it can do so as a result of fragmentation problems
even though the disk as a whole has lots of space --- for one cautionary
example see this thread:

https://www.spinics.net/lists/linux-xfs/msg22856.html

typo-ishly titled "ENSOPC on a 10% used disk". It looks like the XFS crew
installed a fix for the underlying bug ... but that thread is from 2018
and you're running a 2015 Ubuntu release.

Or, to cut to the chase: maybe updating to a less hoary kernel would help.
If you are stuck with this Ubuntu release for some reason, consider using
a less bleeding-edge-at-the-time file system.

regards, tom lane