Help with count(*)

On Friday 14 November 2003 12:51, Rajesh Kumar Mallah wrote:

Hi ,

my database seems to be taking too long for a select count(*)
i think there are lot of dead rows. I do a vacuum full it improves
bu again the performance drops in a short while ,
can anyone please tell me if anything worng with my fsm settings
current fsm=55099264 (not sure how i calculated it)

If you don't need exact count, you can use statistics. Just analyze frequently
and you will get the statistics.

and I didn't exact;y understand this in the text.

INFO:  "profiles": found 0 removable, 369195 nonremovable row versions in
43423 pages
DETAIL:  246130 dead row versions cannot be removed yet.

Is there a transaction holoding up large amount of stuff?

Shridhar

Martha Stewart called it a Good Thing when mallah@trade-india.com (Rajesh Kumar Mallah) wrote:

INFO: "profiles": found 0 removable, 369195 nonremovable row versions in 43423 pages
DETAIL: 246130 dead row versions cannot be removed yet.
Nonremovable row versions range from 136 to 2036 bytes long.

It seems as though you have a transaction open that is holding onto a
whole lot of old rows.

I have seen this happen somewhat-invisibly when a JDBC connection
manager opens transactions for each connection, and then no processing
happens to use those connections for a long time. The open
transactions prevent vacuums from doing any good...
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne&gt; rate me
http://cbbrowne.com/info/multiplexor.html
"Waving away a cloud of smoke, I look up, and am blinded by a bright,
white light. It's God. No, not Richard Stallman, or Linus Torvalds,
but God. In a booming voice, He says: "THIS IS A SIGN. USE LINUX, THE
FREE Unix SYSTEM FOR THE 386." -- Matt Welsh

Christopher Browne kirjutas R, 14.11.2003 kell 16:13:

Martha Stewart called it a Good Thing when mallah@trade-india.com (Rajesh Kumar Mallah) wrote:

INFO: "profiles": found 0 removable, 369195 nonremovable row versions in 43423 pages
DETAIL: 246130 dead row versions cannot be removed yet.
Nonremovable row versions range from 136 to 2036 bytes long.

It seems as though you have a transaction open that is holding onto a
whole lot of old rows.

I have seen this happen somewhat-invisibly when a JDBC connection
manager opens transactions for each connection, and then no processing
happens to use those connections for a long time. The open
transactions prevent vacuums from doing any good...

Can't the backend be made to delay the "real" start of transaction until
the first query gets executed ?

------------
Hannu

Hannu Krosing <hannu@tm.ee> writes:

Christopher Browne kirjutas R, 14.11.2003 kell 16:13:

I have seen this happen somewhat-invisibly when a JDBC connection
manager opens transactions for each connection, and then no processing
happens to use those connections for a long time. The open
transactions prevent vacuums from doing any good...

Can't the backend be made to delay the "real" start of transaction until
the first query gets executed ?

That is on the TODO list. I looked at it briefly towards the end of the
7.4 development cycle, and decided that it was nontrivial and I didn't
have time to make it happen before beta started. I don't recall why it
didn't seem trivial.

regards, tom lane

After a long battle with technology, hannu@tm.ee (Hannu Krosing), an earthling, wrote:

Christopher Browne kirjutas R, 14.11.2003 kell 16:13:

Martha Stewart called it a Good Thing when mallah@trade-india.com (Rajesh Kumar Mallah) wrote:

INFO: "profiles": found 0 removable, 369195 nonremovable row versions in 43423 pages
DETAIL: 246130 dead row versions cannot be removed yet.
Nonremovable row versions range from 136 to 2036 bytes long.

It seems as though you have a transaction open that is holding onto a
whole lot of old rows.

I have seen this happen somewhat-invisibly when a JDBC connection
manager opens transactions for each connection, and then no processing
happens to use those connections for a long time. The open
transactions prevent vacuums from doing any good...

Can't the backend be made to delay the "real" start of transaction until
the first query gets executed ?

One would hope so. Some time when I have the Round Tuits, I ought to
take a browse of the connection pool code to notice if there's
anything to notice.

The thing that I keep imagining would be a slick idea would be to have
a thread periodically go through once for however many connections the
pool permits and fire a short transaction through every
otherwise-unoccupied connection in the pool, in effect, doing a sort
of "vacuum" of the connections. I don't get very favorable reactions
when I suggest that, though...
--
(reverse (concatenate 'string "ac.notelrac.teneerf" "@" "454aa"))
http://cbbrowne.com/info/sgml.html
Rules of the Evil Overlord #80. "If my weakest troops fail to
eliminate a hero, I will send out my best troops instead of wasting
time with progressively stronger ones as he gets closer and closer to
my fortress." <http://www.eviloverlord.com/&gt;

Hannu Krosing wrote:

Christopher Browne kirjutas R, 14.11.2003 kell 16:13:

Martha Stewart called it a Good Thing when mallah@trade-india.com (Rajesh Kumar Mallah) wrote:

INFO: "profiles": found 0 removable, 369195 nonremovable row versions in 43423 pages
DETAIL: 246130 dead row versions cannot be removed yet.
Nonremovable row versions range from 136 to 2036 bytes long.

It seems as though you have a transaction open that is holding onto a
whole lot of old rows.

I have seen this happen somewhat-invisibly when a JDBC connection
manager opens transactions for each connection, and then no processing
happens to use those connections for a long time. The open
transactions prevent vacuums from doing any good...

Can't the backend be made to delay the "real" start of transaction until
the first query gets executed ?

That seems counter intuitive doesn't it? Why write more code in the
server when the client is the thing that has the problem?

Will

Will LaShell <will@lashell.net> writes:

Hannu Krosing wrote:

Can't the backend be made to delay the "real" start of transaction until
the first query gets executed ?

That seems counter intuitive doesn't it? Why write more code in the
server when the client is the thing that has the problem?

Because there are a lot of clients with the same problem :-(

A more principled argument is that we already postpone the setting of
the transaction snapshot until the first query arrives within the
transaction. In a very real sense, the setting of the snapshot *is*
the start of the transaction. So it would make sense if incidental
stuff like VACUUM also thought that the transaction hadn't started
until the first query arrives. (I believe the previous discussion
also agreed that we wanted to postpone the freezing of now(), which
currently also happens at BEGIN rather than the first command after
BEGIN.)

regards, tom lane

On Fri, Nov 14, 2003 at 02:16:56PM -0500, Christopher Browne wrote:

otherwise-unoccupied connection in the pool, in effect, doing a sort
of "vacuum" of the connections. I don't get very favorable reactions
when I suggest that, though...

Because it's a kludge on top of another kludge, perhaps? ;-) This
needs to be fixed properly, not through an ungraceful series of
workarounds.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

On Fri, 14 Nov 2003, Tom Lane wrote:

I believe the previous discussion also agreed that we wanted to postpone
the freezing of now(), which currently also happens at BEGIN rather than
the first command after BEGIN.

Or should that happen at the first call to now()?

/me should ge back and try to find this previous discussion.

--
/Dennis

Tom Lane <tgl@sss.pgh.pa.us> writes:

(I believe the previous discussion also agreed that we wanted to
postpone the freezing of now(), which currently also happens at
BEGIN rather than the first command after BEGIN.)

That doesn't make sense to me: from a user's perspective, the "start
of the transaction" is when the BEGIN is issued, regardless of any
tricks we may play in the backend.

Making now() return the time the current transaction started is
reasonably logical; making now() return "the time when the first
command after the BEGIN in the current transaction was issued" makes a
lot less sense to me.

-Neil

Redirected to -hackers

Neil Conway kirjutas L, 15.11.2003 kell 22:20:

Tom Lane <tgl@sss.pgh.pa.us> writes:

(I believe the previous discussion also agreed that we wanted to
postpone the freezing of now(), which currently also happens at
BEGIN rather than the first command after BEGIN.)

That doesn't make sense to me: from a user's perspective, the "start
of the transaction" is when the BEGIN is issued, regardless of any
tricks we may play in the backend.

For me, the "start of transaction" is not about time, but about grouping
a set of statements into one. So making the exact moment of "start" be
the first statement that actually does something with data seems
perfectly reasonable. If you really need to preserve time, do "select
current_timestamp" and use the result.

Making now() return the time the current transaction started is
reasonably logical; making now() return "the time when the first
command after the BEGIN in the current transaction was issued" makes a
lot less sense to me.

for me "the time the current transactuion is started" == "the time when
the first command after the BEGIN in the current transaction was issued"
and thus I see no conflict here ;)

Delaying the locking effects of transactions as long as possible can
increase performance overall, not just for pathological clients that sit
on idle open transactions.

Probably the latest time we can start the transaction is ath the start
of executor step after the first statement in a transaction is planned
and optimized.

---------------
Hannu

Hannu Krosing <hannu@tm.ee> writes:

Probably the latest time we can start the transaction is ath the start
of executor step after the first statement in a transaction is planned
and optimized.

The transaction has to exist before it can take locks, so the above
would not fly.

A complete example of what we have to think about is:

BEGIN;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
LOCK TABLE foo;
UPDATE foo ... -- or in general a SELECT/UPDATE/INSERT/DELETE query
... etc ...

The transaction snapshot *must* be set at the time of the first query
(here, the UPDATE). It obviously can't be later, and it cannot be
earlier either, because in this sort of example you need the requested
locks to be taken before the snapshot is set.

The transaction must be created (as observed by other backends, in
particular VACUUM) not later than the LOCK statement, else there is
nothing that can own the lock. In principle though, the effects of
BEGIN and perhaps SET could be strictly local to the current backend,
and only when we hit a LOCK or query do we create the transaction
externally.

In practice the problem we observe is clients that issue BEGIN and then
go to sleep (typically because of poorly-designed autocommit behavior in
interface libraries). Postponing externally-visible creation of the
transaction to the first command after BEGIN would be enough to get
around the real-world issues, and it would not require code changes
nearly as extensive as trying to let other stuff like SET happen
"before" the transaction starts.

There isn't any compelling implementation reason when to freeze the
value of now(). Reasonable options are
1. at BEGIN (current behavior)
2. at transaction's external creation
3. at freezing of transaction snapshot
#1 and #2 are actually the same at the moment, but could be decoupled
as sketched above, in which case the behavior of #2 would effectively
become "at first command afte BEGIN".

In the previous thread:
http://archives.postgresql.org/pgsql-hackers/2003-03/msg01178.php
I argued that now() should be frozen at the time of the transaction
snapshot, and I still think that that's a defensible behavior.

regards, tom lane

Neil Conway <neilc@samurai.com> writes:

Tom Lane <tgl@sss.pgh.pa.us> writes:

(I believe the previous discussion also agreed that we wanted to
postpone the freezing of now(), which currently also happens at
BEGIN rather than the first command after BEGIN.)

That doesn't make sense to me: from a user's perspective, the "start
of the transaction" is when the BEGIN is issued, regardless of any
tricks we may play in the backend.

That's defensible when the user issued the BEGIN himself. When the
BEGIN is coming from some interface library's autocommit logic, it's
a lot less defensible. If you consult the archives, you will find
actual user complaints about "why is now() returning a very old time?"
that we traced to use of interface layers that handle "commit()" by
issuing "COMMIT; BEGIN;".

When BEGIN actually is issued by live application logic, I'd expect it
to be followed immediately by some kind of command --- so the user would
be unable to tell the difference in practice.

Hannu moved this thread to -hackers, please follow up there if you want
to discuss it more.

regards, tom lane

On Sun, 16 Nov 2003, Tom Lane wrote:

There isn't any compelling implementation reason when to freeze the
value of now(). Reasonable options are
1. at BEGIN (current behavior)
2. at transaction's external creation
3. at freezing of transaction snapshot
#1 and #2 are actually the same at the moment, but could be decoupled
as sketched above, in which case the behavior of #2 would effectively
become "at first command afte BEGIN".

I argued that now() should be frozen at the time of the transaction
snapshot, and I still think that that's a defensible behavior.

Is it important exactly what value is returned as long as it's the same in
the whole transaction? I think not.

To me it would be just as logical to fix it at the first call to now() in
the transaction. The first time you call it you get the actual time as it
is now and the next time you get the same as before since every operation
in the transaction logically happens at the same time. If you don't call
now() at all, the system time will not be fetched at all.

--
/Dennis

Hannu Krosing <hannu@tm.ee> writes:

For me, the "start of transaction" is not about time, but about grouping
a set of statements into one. So making the exact moment of "start" be
the first statement that actually does something with data seems
perfectly reasonable.

This might be a perfectly logical change in semantics, but what
benefit does it provide over the old way of doing things?

What does BEGIN actually do now, from a user's perspective? At
present, it "starts a transaction block", which is pretty simple. If
we adopted the proposed change, it would "change the state of the
system so that the next command is part of a new transaction". This is
naturally more complex; but more importantly, what benefit does it
ACTUALLY provide to the user?

(I can't see one, but perhaps I'm missing something...)

Delaying the locking effects of transactions as long as possible can
increase performance overall, not just for pathological clients that sit
on idle open transactions.

I agree, but this is irrelevant to the semantics of now().

-Neil

Tom Lane <tgl@sss.pgh.pa.us> writes:

That's defensible when the user issued the BEGIN himself. When the
BEGIN is coming from some interface library's autocommit logic, it's
a lot less defensible. If you consult the archives, you will find
actual user complaints about "why is now() returning a very old time?"
that we traced to use of interface layers that handle "commit()" by
issuing "COMMIT; BEGIN;".

Hmmm... I agree this behavior isn't ideal, although I can see the case
for viewing this as a mistake by the application developer: they are
assuming that they know exactly when transactions begin, which is not
a feature provided by their language interface. They should be using
current_timestamp, and/or changing their language interface's
configuration.

That said, I think this is a minor irritation at best. The dual
drawbacks of breaking backward compatibility and making the BEGIN
semantics more confusing is enough to leave me satisfies with the
status quo.

If we do change this, I think Dennis' idea of making now() always
return the same value within a given transaction is interesting: that
might be a way to fix this problem without confusing the semantics of
BEGIN.

-Neil

Neil Conway <neilc@samurai.com> writes:

Hmmm... I agree this behavior isn't ideal, although I can see the case
for viewing this as a mistake by the application developer: they are
assuming that they know exactly when transactions begin, which is not
a feature provided by their language interface.

Well, actually, it's a bug in the interface IMHO. But as I said in the
last thread, it's a fairly widespread bug. We've been taking the
position that the interface libraries should get fixed, and that's not
happening. It's probably time to look at a server-side fix.

If we do change this, I think Dennis' idea of making now() always
return the same value within a given transaction is interesting:

You mean the time of the first now() call? I thought that was an
interesting idea also, but it's probably not going to look so hot
when we complete the TODO item of adding access to
the start-of-current-statement time. Having start-of-transaction be
later than start-of-statement isn't gonna fly :-(. If we were willing
to abandon that TODO item then I'd be interested in defining now() as
Dennis suggested.

regards, tom lane

Tom Lane wrote:

Neil Conway <neilc@samurai.com> writes:

Hmmm... I agree this behavior isn't ideal, although I can see the case
for viewing this as a mistake by the application developer: they are
assuming that they know exactly when transactions begin, which is not
a feature provided by their language interface.

Well, actually, it's a bug in the interface IMHO. But as I said in the
last thread, it's a fairly widespread bug. We've been taking the
position that the interface libraries should get fixed, and that's not
happening. It's probably time to look at a server-side fix.

If we do change this, I think Dennis' idea of making now() always
return the same value within a given transaction is interesting:

You mean the time of the first now() call? I thought that was an
interesting idea also, but it's probably not going to look so hot
when we complete the TODO item of adding access to
the start-of-current-statement time. Having start-of-transaction be
later than start-of-statement isn't gonna fly :-(. If we were willing
to abandon that TODO item then I'd be interested in defining now() as
Dennis suggested.

Defining now() as the first call seems pretty arbitrary to me. I can't
think of any time-based interface that has that API. And what if a
trigger called now() in an earlier query and you didn't even know about
it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Neil Conway <neilc@samurai.com> writes:

What does BEGIN actually do now, from a user's perspective?

I think you're thinking about this all wrong. BEGIN doesn't "do" anything.
It's not a procedural statement, it's a declaration. It declares that the
block of statements form a transaction so reads should be consistent and
failures should be handled in a particular way to preserve data integrity.

Given that declaration and the guarantees it requires of the database it's
then up to the database to figure out what constraints that imposes on what
the database can do and still meet the guarantees the BEGIN declaration
requires. The more clever the database is about minimizing those restrictions
the better as it means the database can run more efficiently.

For what it's worth, this is how Oracle handles things too. On the
command-line issuing a BEGIN following a COMMIT is just noise; you're _always_
in a transaction. A COMMIT ends the previous the transaction and implicitly
starts the next transaction. But the snapshot isn't frozen until you first
read from a table.

I'm not sure what other databases do, but I think this is why clients behave
like this. They think of BEGIN as a declaration and therefore initiating a
COMMIT;BEGIN; at the end of every request is perfectly logical, and works fine
in at least Oracle, and probably other databases.

--
greg

On Sun, 17 Nov 2003, Greg Stark wrote:

Neil Conway <neilc@samurai.com> writes:

What does BEGIN actually do now, from a user's perspective?

I think you're thinking about this all wrong. BEGIN doesn't "do" anything.
It's not a procedural statement, it's a declaration. It declares that the
block of statements form a transaction so reads should be consistent and
failures should be handled in a particular way to preserve data integrity.

Given that declaration and the guarantees it requires of the database it's
then up to the database to figure out what constraints that imposes on what
the database can do and still meet the guarantees the BEGIN declaration
requires. The more clever the database is about minimizing those restrictions
the better as it means the database can run more efficiently.

For what it's worth, this is how Oracle handles things too. On the
command-line issuing a BEGIN following a COMMIT is just noise; you're _always_
in a transaction. A COMMIT ends the previous the transaction and implicitly
starts the next transaction. But the snapshot isn't frozen until you first
read from a table.

The earlier portion of the described behavior is AFAICS not complient to
SQL99 at least. COMMIT (without AND CHAIN) terminates a transaction and
does not begin a new one. The new transaction does not begin until a
transaction initiating command (for example START TRANSACTION, CREATE
TABLE, INSERT, ...) is executed. The set of things you can do that aren't
initiating is fairly small admittedly, but it's not a null set.

Tom Lane kirjutas E, 17.11.2003 kell 02:08:

Neil Conway <neilc@samurai.com> writes:

Hmmm... I agree this behavior isn't ideal, although I can see the case
for viewing this as a mistake by the application developer: they are
assuming that they know exactly when transactions begin, which is not
a feature provided by their language interface.

Well, actually, it's a bug in the interface IMHO. But as I said in the
last thread, it's a fairly widespread bug.

I'm not sure that it is a client-side bug. For example Oracle seems to
_always_ have a transaction going, i.e. you can't be "outside" of
transaction, and you use just COMMIT to commit old _and_start_new_
transaction.

IIRC the same is true for DB2.

For these database the BEGIN TRANSACTION command is mainly used for
starting nested transactions, which we don't have.

We've been taking the
position that the interface libraries should get fixed, and that's not
happening. It's probably time to look at a server-side fix.

Maybe "fixing" the interface libraries would make them incompatible with
*DBC's for all other databases in some subtle ways ?

-----------------
Hannu

Bruce Momjian kirjutas E, 17.11.2003 kell 02:31:

Defining now() as the first call seems pretty arbitrary to me. I can't
think of any time-based interface that has that API. And what if a
trigger called now() in an earlier query and you didn't even know about
it.

That would be OK. The whole point of that previous discussion was to
have now() that returns the same value over the span of the whole
transaction.

It would be even better to have now() that returns the time current
transaction is COMMITted as this is the time other backend become aware
of it ;)

-----------
Hannu

Hannu Krosing wrote:

Tom Lane kirjutas E, 17.11.2003 kell 02:08:

Neil Conway <neilc@samurai.com> writes:

Hmmm... I agree this behavior isn't ideal, although I can see the case
for viewing this as a mistake by the application developer: they are
assuming that they know exactly when transactions begin, which is not
a feature provided by their language interface.

Well, actually, it's a bug in the interface IMHO. But as I said in the
last thread, it's a fairly widespread bug.

I'm not sure that it is a client-side bug. For example Oracle seems to
_always_ have a transaction going, i.e. you can't be "outside" of
transaction, and you use just COMMIT to commit old _and_start_new_
transaction.

IIRC the same is true for DB2.

Actually, in oracle a new transaction starts with first DDL after a commit. That
does not include DML BTW.

And Damn.. Actually I recently fixed a "bug" where I had to force a start of
transaction in Pro*C, immediately after commit. Otherwise a real start of
transaction could be anywhere down the line, causing some weird concurrency
issues. Rather than fiddling with oracle support, I would hack my source code,
especially this is not the first oracle bug I have worked around....:-(

The fact that I couldn't control exact transaction start was such a irritation
to put it mildly.. I sooooo missed 'exec sql begin work' in ecpg..:-)

We've been taking the
position that the interface libraries should get fixed, and that's not
happening. It's probably time to look at a server-side fix.

I hope that does not compramise transaction control I have with libpq/ecpg etc.

And when we are talking about interface libraries, how many of them are within
PG control and how many are not? With languages maintenend by postgresql group,
it should behave correctly, right? E.g pl/perl,pl/python etc.

And for other interface libraries, what are they exactly? php? Can't we just
send them a stinker/patch to get that damn thing right(Whatever wrong they are
doing. I have kinda lost thread on it..:-) Was it exact time of transaction
start v/s now()?)

Shridhar

Hannu Krosing wrote:

Bruce Momjian kirjutas E, 17.11.2003 kell 02:31:

Defining now() as the first call seems pretty arbitrary to me. I can't
think of any time-based interface that has that API. And what if a
trigger called now() in an earlier query and you didn't even know about
it.

That would be OK. The whole point of that previous discussion was to
have now() that returns the same value over the span of the whole
transaction.

I think my issue is that there isn't any predictable way for a user to
know when the now() time is recorded. By using start of transaction, at
least we know for sure the point in time it is showing.

It would be even better to have now() that returns the time current
transaction is COMMITted as this is the time other backend become aware
of it ;)

True, but implementing that would be very hard.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Hannu Krosing wrote:

It would be even better to have now() that returns the time current
transaction is COMMITted as this is the time other backend become aware
of it ;)

True, but implementing that would be very hard.

Son, that was a *joke* ...

regards, tom lane