What is the tuplestore?

What if this error message pertains to something happening on the
application side?

Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:

Hi,

v9.6.6

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore
temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what "tuplestore"
is
(Google doesn't help), and #2 there's lots of space on all my file
systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Thanks

--
Angular momentum makes the world go 'round.

--
El genio es 1% inspiración y 99% transpiración.
Thomas Alva Edison
http://pglearn.blogspot.mx/

There's certainly a problem with the application, but the error is in the
pg_log, not the application log.

On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:

What if this error message pertains to something happening on the
application side?

Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com
<mailto:ronljohnsonjr@gmail.com>>:

Hi,

v9.6.6

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR:  could not write to tuplestore
temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what
"tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file
systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Thanks

--
Angular momentum makes the world go 'round.

Maybe the temp space got released right after the failure?
https://grokbase.com/t/postgresql/pgsql-general/02ag7k8gcr/tuplestore-write-failed
do you have space usage charts for that partition? doesn't it show a spike
during that time?

Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:

There's certainly a problem with the application, but the error is in the
pg_log, not the application log.

On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:

What if this error message pertains to something happening on the
application side?

Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:

Hi,

v9.6.6

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore
temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what "tuplestore"
is
(Google doesn't help), and #2 there's lots of space on all my file
systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Thanks

--
Angular momentum makes the world go 'round.

--
El genio es 1% inspiración y 99% transpiración.
Thomas Alva Edison
http://pglearn.blogspot.mx/

Which file system (specifically, which directory)?  Is it
data/base/pgsql_tmp?  There's 96GB free, which is 74% of the volume.

On 12/10/2018 04:50 PM, Rene Romero Benavides wrote:

Maybe the temp space got released right after the failure?
https://grokbase.com/t/postgresql/pgsql-general/02ag7k8gcr/tuplestore-write-failed
do you have space usage charts for that partition? doesn't it show a spike
during that time?

Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com
<mailto:ronljohnsonjr@gmail.com>>:

There's certainly a problem with the application, but the error is in
the pg_log, not the application log.

On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:

What if this error message pertains to something happening on the
application side?

Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron
<ronljohnsonjr@gmail.com <mailto:ronljohnsonjr@gmail.com>>:

Hi,

v9.6.6

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW
13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR:  could not write to
tuplestore temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what
"tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my
file systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Thanks

--
Angular momentum makes the world go 'round.

Yes, pgsql_tmp, you ought to be looking for a sudden and drastic jump in
space utilization around the time of the error message. You're not
concerned with the current space utilization, but with the one around that
time, because, it probably got freed right after the error was raised.
How many times has this happened ? what kind of queries were running at
that time? can you identify something that could have required lots of
temporary space?

Am Mo., 10. Dez. 2018 um 17:33 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:

Which file system (specifically, which directory)? Is it
data/base/pgsql_tmp? There's 96GB free, which is 74% of the volume.

On 12/10/2018 04:50 PM, Rene Romero Benavides wrote:

Maybe the temp space got released right after the failure?

https://grokbase.com/t/postgresql/pgsql-general/02ag7k8gcr/tuplestore-write-failed
do you have space usage charts for that partition? doesn't it show a spike
during that time?

Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:

There's certainly a problem with the application, but the error is in the
pg_log, not the application log.

On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:

What if this error message pertains to something happening on the
application side?

Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:

Hi,

v9.6.6

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748
SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore
temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what "tuplestore"
is
(Google doesn't help), and #2 there's lots of space on all my file
systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Thanks

--
Angular momentum makes the world go 'round.

--
El genio es 1% inspiración y 99% transpiración.
Thomas Alva Edison
http://pglearn.blogspot.mx/

On Tue, Dec 11, 2018 at 2:56 AM Ron <ronljohnsonjr@gmail.com> wrote:

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Maybe the setting log_temp_files could help you identify the query
that is doing it?

https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-TEMP-FILES

--
Thomas Munro
http://www.enterprisedb.com

I extracted the queries from pg_log and sent them to the customer team.  To
fill up 96GB of disk space seems like the customer selected a huge date range.

On 12/10/2018 06:07 PM, Rene Romero Benavides wrote:

Yes, pgsql_tmp, you ought to be looking for a sudden and drastic jump in
space utilization around the time of the error message. You're not
concerned with the current space utilization, but with the one around that
time, because, it probably got freed right after the error was raised.
How many times has this happened ? what kind of queries were running at
that time? can you identify something that could have required lots of
temporary space?

Am Mo., 10. Dez. 2018 um 17:33 Uhr schrieb Ron <ronljohnsonjr@gmail.com
<mailto:ronljohnsonjr@gmail.com>>:

Which file system (specifically, which directory)?  Is it
data/base/pgsql_tmp?  There's 96GB free, which is 74% of the volume.

On 12/10/2018 04:50 PM, Rene Romero Benavides wrote:

Maybe the temp space got released right after the failure?
https://grokbase.com/t/postgresql/pgsql-general/02ag7k8gcr/tuplestore-write-failed
do you have space usage charts for that partition? doesn't it show a
spike during that time?

Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron
<ronljohnsonjr@gmail.com <mailto:ronljohnsonjr@gmail.com>>:

There's certainly a problem with the application, but the error
is in the pg_log, not the application log.

On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:

What if this error message pertains to something happening on
the application side?

Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron
<ronljohnsonjr@gmail.com <mailto:ronljohnsonjr@gmail.com>>:

Hi,

v9.6.6

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW
13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR:  could not write to
tuplestore temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what
"tuplestore" is
(Google doesn't help), and #2 there's lots of space on all
my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Thanks

--
Angular momentum makes the world go 'round.

--
El genio es 1% inspiración y 99% transpiración.
Thomas Alva Edison
http://pglearn.blogspot.mx/

--
Angular momentum makes the world go 'round.

On 12/10/2018 06:24 PM, Thomas Munro wrote:

On Tue, Dec 11, 2018 at 2:56 AM Ron <ronljohnsonjr@gmail.com> wrote:

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Maybe the setting log_temp_files could help you identify the query
that is doing it?

https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-TEMP-FILES

Great idea; I've made the change.  What do I grep for in the pg_log file?

--
Angular momentum makes the world go 'round.

On Tue, Dec 11, 2018 at 3:58 PM Ron <ronljohnsonjr@gmail.com> wrote:

On 12/10/2018 06:24 PM, Thomas Munro wrote:

On Tue, Dec 11, 2018 at 2:56 AM Ron <ronljohnsonjr@gmail.com> wrote:

2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device

I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)

Maybe the setting log_temp_files could help you identify the query
that is doing it?

https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-TEMP-FILES

Great idea; I've made the change. What do I grep for in the pg_log file?

It should say something like:

LOG: temporary file: path "base/pgsql_tmp/...", size ...
STATEMENT: ...

--
Thomas Munro
http://www.enterprisedb.com

When one get a "No space left on device" and there is a lot of space it is
sometimes caused by lack of inodes.

Try run the command:
df --inodes

On Mon, Dec 10, 2018 at 4:56 PM Ron <ronljohnsonjr@gmail.com> wrote: