Wierd performance issue with 8.1cvs

On Wed, Apr 20, 2005 at 11:09:19AM -0700, Josh Berkus wrote:

At OSDL we're seeing a wierd performance crash on 8.1cvs. What's wierd about
it is that it doesn't happen all the time -- about 1 out of 4 test runs.
What it looks like happens sometimes is that performance drops dramatically
at the first checkpoint, and never comes back. But there's oprofiles and
things to make a more insightful analysis:

What is the "wait" line in the Processor Utilization graphic? Does it
have anything to do with spinlocks? Was it so high in 8.0 tests?

--
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
Jude: I wish humans laid eggs
Ringlord: Why would you want humans to lay eggs?
Jude: So I can eat them

Josh Berkus <josh@agliodbs.com> writes:

Folks,

At OSDL we're seeing a wierd performance crash on 8.1cvs. What's wierd about
it is that it doesn't happen all the time -- about 1 out of 4 test runs.
What it looks like happens sometimes is that performance drops dramatically
at the first checkpoint, and never comes back. But there's oprofiles and
things to make a more insightful analysis:

3 test runs exhibit it:
http://khack.osdl.org/stp/301531/0.html
http://khack.osdl.org/stp/301736/0.html
http://khack.osdl.org/stp/301730/0.html

That dropoff at 60 minutes is the *first* checkpoint?! On an 80m test run?

That's a totally unrealistic configuration. Do you have any reason to think
the drop-off isn't just because all that pending i/o that you've postponed for
so long is finally having to get written out? Worse, it's forcing Postgres to
fsync files after 60m of i/o has been performed, flushing huge queues of i/o.

The benchmarks performed in this configuration are completely bogus. They
aren't including the time to checkpoint the last 20m of i/o, a quarter of all
the i/o in the test.

You really have to lower the checkpoint timeout to something realistic, like
5m or so. Otherwise these tests are just useless.

--
greg

Josh Berkus <josh@agliodbs.com> writes:

At OSDL we're seeing a wierd performance crash on 8.1cvs.

Permit me a digression for a pet peeve...

"Weird" is spelled "weird". Not "wierd".

Yes, I know the nursery rhyme as well as you do --- i before e except
after c, etc etc. But weird is spelled weirdly. Appropriate, isn't it?

regards, tom lane

PS: If you need an authoritative reference, will the OED do?
http://www.askoxford.com/concise_oed/weird?view=uk

Permit me a digression for a pet peeve...

"Weird" is spelled "weird". Not "wierd".

Yes, I know the nursery rhyme as well as you do --- i before e except
after c, etc etc. But weird is spelled weirdly. Appropriate, isn't it?

It's also my pet peeve, but I long ago stopped bothering to correct people.

One may also wish to consider "feisty" :)

Chris

On Sun, 24 Apr 2005, Christopher Kings-Lynne wrote:

Permit me a digression for a pet peeve...

"Weird" is spelled "weird". Not "wierd".

Yes, I know the nursery rhyme as well as you do --- i before e except
after c, etc etc. But weird is spelled weirdly. Appropriate, isn't it?

It's also my pet peeve, but I long ago stopped bothering to correct people.

I think it's quite useful to correct, because many of us use english only
when reading mailing lists and documentation :)

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov wrote:

I think it's quite useful to correct, because many of us use english only
when reading mailing lists and documentation :)

I think that it's important to refrain from corrections on a public
forum as long as the essence of the message is clear. Some people might
get offended, others might just stop writing because they get
intimidated by the seemingly high demands on correct spelling or
sentence structure.

I appreciate getting corrected by people I know in a limited forum. I
would not expect it when I do a mistakes here. Can't say it ever has
happend although there's often good grounds for it so I have nothing to
complain about. :-)

Regards,
Thomas Hallgren

I appreciate getting corrected by people I know in a limited
forum. I would not expect it when I do a mistakes here. Can't
say it ever has happend although there's often good grounds
for it so I have nothing to complain about. :-)

I think you meant to say 'I Can't say it ever has happened...'

:)

... John

On Sun, 24 Apr 2005, Thomas Hallgren wrote:

Oleg Bartunov wrote:

I think it's quite useful to correct, because many of us use english only
when reading mailing lists and documentation :)

I think that it's important to refrain from corrections on a public forum as
long as the essence of the message is clear. Some people might get offended,
others might just stop writing because they get intimidated by the seemingly
high demands on correct spelling or sentence structure.

I appreciate getting corrected by people I know in a limited forum. I would
not expect it when I do a mistakes here. Can't say it ever has happend
although there's often good grounds for it so I have nothing to complain
about. :-)

It's difficult to opposed you but "limited forum" is not we could afford.
When I see a lot of mistakes, crude words in russian forums I always feel
myself uncomfortable. Unfortunately, I can't imagine how it looks for
english speaking people. Of course, there are simple misprints which could
be painlessly ommited, but I see nothing offending against correcting
gross mistakes. Usually, people just too busy to notice them.

Regards,
Thomas Hallgren

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov wrote:

On Sun, 24 Apr 2005, Thomas Hallgren wrote:

Oleg Bartunov wrote:

I think it's quite useful to correct, because many of us use
english only
when reading mailing lists and documentation :)

I think that it's important to refrain from corrections on a public
forum as long as the essence of the message is clear. Some people
might get offended, others might just stop writing because they get
intimidated by the seemingly high demands on correct spelling or
sentence structure.

I appreciate getting corrected by people I know in a limited forum. I
would not expect it when I do a mistakes here. Can't say it ever has
happend although there's often good grounds for it so I have nothing
to complain about. :-)

It's difficult to opposed you but "limited forum" is not we could afford.
When I see a lot of mistakes, crude words in russian forums I always feel
myself uncomfortable. Unfortunately, I can't imagine how it looks for
english speaking people. Of course, there are simple misprints which
could
be painlessly ommited, but I see nothing offending against correcting
gross mistakes. Usually, people just too busy to notice them.

This Sentience has tree errors.

Regards,
Thomas Hallgren

Guys,

"Weird" is spelled "weird". Not "wierd".

OK, spelling errors taken into account. Now could we perhaps address the
**postgresql** errors?

I'm seeing this kind of "performance plunge" on 8.1cvs in one of every 3 runs.
It's obviously a serious stability issue, whatever is causing it.

--
Josh Berkus
Aglio Database Solutions
San Francisco

Josh Berkus <josh@agliodbs.com> writes:

I'm seeing this kind of "performance plunge" on 8.1cvs in one of every 3 runs.
It's obviously a serious stability issue, whatever is causing it.

I concur with the upthread suggestion that it may come from not doing
checkpoints in a realistic fashion, thereby allowing too much queued
I/O work to build up. What happens if you set the checkpoint interval
to 5 or 10 minutes?

regards, tom lane

Tom,

I concur with the upthread suggestion that it may come from not doing
checkpoints in a realistic fashion, thereby allowing too much queued
I/O work to build up.  What happens if you set the checkpoint interval
to 5 or 10 minutes?

I'll test. Keep in mind that it takes me a couple of days to run a new test
(I've got 11 in the queue right now) so it'll be a bit.

--
Josh Berkus
Aglio Database Solutions
San Francisco