pgbench -i spends all its time doing CHECKPOINT

Tom Lane wrote:

Could we remove lines 552-560 of pgbench.c? The behavior that guarded
against is long gone, and forcing a checkpoint every few thousand tuples
seems to be putting a huge crimp in the speed of pgbench -i ...

I don't see any need for it. It isn't even needed for 7.1.3.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

Could we remove lines 552-560 of pgbench.c? The behavior that guarded
against is long gone, and forcing a checkpoint every few thousand tuples
seems to be putting a huge crimp in the speed of pgbench -i ...

I don't see any need for it.

Au contraire. I'm nearly done with pgbench -i -s 500 after about an
hour. The unmodified code ran for several hours and was less than
half done.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

Could we remove lines 552-560 of pgbench.c? The behavior that guarded
against is long gone, and forcing a checkpoint every few thousand tuples
seems to be putting a huge crimp in the speed of pgbench -i ...

I don't see any need for it.

Au contraire. I'm nearly done with pgbench -i -s 500 after about an
hour. The unmodified code ran for several hours and was less than
half done.

I meant I don't see any need for that checkpoint code.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

I meant I don't see any need for that checkpoint code.

Oh, my mistake; I thought you meant you didn't want to make a change
now.

regards, tom lane

Could we remove lines 552-560 of pgbench.c? The behavior that guarded
against is long gone, and forcing a checkpoint every few thousand tuples
seems to be putting a huge crimp in the speed of pgbench -i ...

Yup. Maybe we could ifdef'ed out until we implement true UNDO...
--
Tatsuo Ishii

Tatsuo Ishii wrote:

Could we remove lines 552-560 of pgbench.c? The behavior that guarded
against is long gone, and forcing a checkpoint every few thousand tuples
seems to be putting a huge crimp in the speed of pgbench -i ...

Yup. Maybe we could ifdef'ed out until we implement true UNDO...

I think we should just remove it. The idea that we are going to do UNDO
which allows unlimited log file growth for long transactions seems like
a loser to me.

Actually, that brings up a question I had. In 7.1.0, we didn't recycle
WAL segements that were used by open transactions during CHECKPOINT,
while in 7.1.3 and later, we do recycle them after CHECKPOINT. My
question is if we do a big transaction that needs 10 log segments, do we
force an early CHECKPOINT to clear out the WAL segments or do we just
wait for the proper interval?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

question is if we do a big transaction that needs 10 log segments, do we
force an early CHECKPOINT to clear out the WAL segments or do we just
wait for the proper interval?

A checkpoint is forced after every CHECKPOINT_SEGMENTS log segments,
regardless of longevity of transactions. See
http://developer.postgresql.org/docs/postgres/wal-configuration.html

Since segments before the checkpoint-before-last are deleted or recycled
after each checkpoint, the maximum number of back segments would
normally be 2 * CHECKPOINT_SEGMENTS. We also pre-create WAL_FILES
future log segments. Counting the current segment gives a total of
WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments.

AFAICS, the only way to force the current code into creating more than
WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments is to be generating
WAL entries at such a high rate that more than WAL_FILES log segments
are filled before a triggered checkpoint can be completed.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

question is if we do a big transaction that needs 10 log segments, do we
force an early CHECKPOINT to clear out the WAL segments or do we just
wait for the proper interval?

A checkpoint is forced after every CHECKPOINT_SEGMENTS log segments,
regardless of longevity of transactions. See
http://developer.postgresql.org/docs/postgres/wal-configuration.html

Since segments before the checkpoint-before-last are deleted or recycled
after each checkpoint, the maximum number of back segments would
normally be 2 * CHECKPOINT_SEGMENTS. We also pre-create WAL_FILES
future log segments. Counting the current segment gives a total of
WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments.

AFAICS, the only way to force the current code into creating more than
WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments is to be generating
WAL entries at such a high rate that more than WAL_FILES log segments
are filled before a triggered checkpoint can be completed.

Very interesting. Thanks. Is there a reason someone would manually run
the CHECKPOINT command?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026