Query never returns ...

Followup:

This query, run against the same dataset in a MS Access 2000 database,
returns immediately with the resultset. (I truly mean immediately) The
computer running the Access db is a laptop, PII 266 w/ 128MB RAM (an old
Dell Latitude). The server running PostgreSQL is a dual PIII 450 w/
256MB RAM.

This is just meant as an FYI.

-Brice

Brice Ruth wrote:

The following query:

SELECT
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.MedCondID,
tblMedCond.PatientName AS MedCondPatientName,
tblMedCond.ProfessionalName AS MedCondProfessionalName,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel
FROM
tblSIDEDrugLink,
tblMedCond
WHERE
(tblSIDEDrugLink.DrugID IN ('DG-18698')) AND
(tblSIDEDrugLink.MedCondID = tblMedCond.MedCondID)
ORDER BY
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel,
tblSIDEDrugLink.MedCondID;

seems to not be liked by PostgreSQL. Table 'tblSIDEDrugLink' has the
following structure:

CREATE TABLE TBLSIDEDRUGLINK
(
DRUGID VARCHAR(10) NOT NULL,
MEDCONDID VARCHAR(10) NOT NULL,
FREQUENCY INT2,
SEVERITYLEVEL INT2,
CONSTRAINT PK_TBLSIDEDRUGLINK PRIMARY KEY (DRUGID, MEDCONDID)
);

with the following index:
CREATE INDEX IX_TBLSIDEDRUGLINK1 ON TBLSIDEDRUGLINK (MEDCONDID);

This table has 153,288 rows.

Table 'tblMedCond' has the following structure:

CREATE TABLE TBLMEDCOND
(
MEDCONDID VARCHAR(10) NOT NULL,
PROFESSIONALNAME VARCHAR(58),
PATIENTNAME VARCHAR(58),
CONSTRAINT PK_TBLMEDCOND PRIMARY KEY (MEDCONDID)
);

This table has 1,730 rows.

The query above is made by a third-party API that I don't have the
source for, so I can't modify the query in the API, though the
third-party has been quite willing to help out - they may even ship me a
'special' version of the API if there's something in this query that
PostgreSQL for some reason doesn't implement efficiently enough.

If it would help anyone to see the query plan or such - I can modify the
logs to show that, just let me know.

Btw - I've let this query run for a while & I haven't seen it complete
... soooo ... I don't know if it would ever complete or not.

Any help at all is as always, appreciated.

Sincerest regards,
--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

FYI - I let the query run for 11.5 minutes before killing it off. It
had consumed 11 minutes, 18 seconds of CPU time (reported by ps). The
following messages are from the server log, I'm pasting in all messages
directly following the query:

010208.10:04:29.473 [24041] ProcessQuery
010208.10:15:59.212 [24041] FATAL 1: The system is shutting down
010208.10:15:59.213 [24041] AbortCurrentTransaction

FATAL: s_lock(4001600c) at spin.c:111, stuck spinlock. Aborting.

FATAL: s_lock(4001600c) at spin.c:111, stuck spinlock. Aborting.
Server process (pid 24041) exited with status 134 at Thu Feb 8 10:17:09 2001
Terminating any active server processes...
Server processes were terminated at Thu Feb 8 10:17:09 2001

Regards,
Brice Ruth

Brice Ruth wrote:

The following query:

SELECT
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.MedCondID,
tblMedCond.PatientName AS MedCondPatientName,
tblMedCond.ProfessionalName AS MedCondProfessionalName,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel
FROM
tblSIDEDrugLink,
tblMedCond
WHERE
(tblSIDEDrugLink.DrugID IN ('DG-18698')) AND
(tblSIDEDrugLink.MedCondID = tblMedCond.MedCondID)
ORDER BY
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel,
tblSIDEDrugLink.MedCondID;

seems to not be liked by PostgreSQL. Table 'tblSIDEDrugLink' has the
following structure:

CREATE TABLE TBLSIDEDRUGLINK
(
DRUGID VARCHAR(10) NOT NULL,
MEDCONDID VARCHAR(10) NOT NULL,
FREQUENCY INT2,
SEVERITYLEVEL INT2,
CONSTRAINT PK_TBLSIDEDRUGLINK PRIMARY KEY (DRUGID, MEDCONDID)
);

with the following index:
CREATE INDEX IX_TBLSIDEDRUGLINK1 ON TBLSIDEDRUGLINK (MEDCONDID);

This table has 153,288 rows.

Table 'tblMedCond' has the following structure:

CREATE TABLE TBLMEDCOND
(
MEDCONDID VARCHAR(10) NOT NULL,
PROFESSIONALNAME VARCHAR(58),
PATIENTNAME VARCHAR(58),
CONSTRAINT PK_TBLMEDCOND PRIMARY KEY (MEDCONDID)
);

This table has 1,730 rows.

The query above is made by a third-party API that I don't have the
source for, so I can't modify the query in the API, though the
third-party has been quite willing to help out - they may even ship me a
'special' version of the API if there's something in this query that
PostgreSQL for some reason doesn't implement efficiently enough.

If it would help anyone to see the query plan or such - I can modify the
logs to show that, just let me know.

Btw - I've let this query run for a while & I haven't seen it complete
... soooo ... I don't know if it would ever complete or not.

Any help at all is as always, appreciated.

Sincerest regards,
--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

What does explain show for the query and have you run
vacuum analyze recently on the tables?

On Thu, 8 Feb 2001, Brice Ruth wrote:

Stephan,

Here is what EXPLAIN shows:

NOTICE: QUERY PLAN:

Sort (cost=0.02..0.02 rows=1 width=64)
-> Nested Loop (cost=0.00..0.01 rows=1 width=64)
-> Seq Scan on tblmedcond (cost=0.00..0.00 rows=1 width=36)
-> Seq Scan on tblsidedruglink (cost=0.00..0.00 rows=1 width=28)

As for vacuum analyze - prior to running into these problems, I deleted
all data from the database (using delete from <tblname>) and then ran
vacuumdb -a, after which I loaded the data into the tables using 'copy
... from' - there have been no updates to the database since then -
merely selects.

-Brice

Stephan Szabo wrote:

What does explain show for the query and have you run
vacuum analyze recently on the tables?

On Thu, 8 Feb 2001, Brice Ruth wrote:

The following query:

SELECT
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.MedCondID,
tblMedCond.PatientName AS MedCondPatientName,
tblMedCond.ProfessionalName AS MedCondProfessionalName,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel
FROM
tblSIDEDrugLink,
tblMedCond
WHERE
(tblSIDEDrugLink.DrugID IN ('DG-18698')) AND
(tblSIDEDrugLink.MedCondID = tblMedCond.MedCondID)
ORDER BY
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel,
tblSIDEDrugLink.MedCondID;

seems to not be liked by PostgreSQL. Table 'tblSIDEDrugLink' has the
following structure:

CREATE TABLE TBLSIDEDRUGLINK
(
DRUGID VARCHAR(10) NOT NULL,
MEDCONDID VARCHAR(10) NOT NULL,
FREQUENCY INT2,
SEVERITYLEVEL INT2,
CONSTRAINT PK_TBLSIDEDRUGLINK PRIMARY KEY (DRUGID, MEDCONDID)
);

with the following index:
CREATE INDEX IX_TBLSIDEDRUGLINK1 ON TBLSIDEDRUGLINK (MEDCONDID);

This table has 153,288 rows.

Table 'tblMedCond' has the following structure:

CREATE TABLE TBLMEDCOND
(
MEDCONDID VARCHAR(10) NOT NULL,
PROFESSIONALNAME VARCHAR(58),
PATIENTNAME VARCHAR(58),
CONSTRAINT PK_TBLMEDCOND PRIMARY KEY (MEDCONDID)
);

This table has 1,730 rows.

The query above is made by a third-party API that I don't have the
source for, so I can't modify the query in the API, though the
third-party has been quite willing to help out - they may even ship me a
'special' version of the API if there's something in this query that
PostgreSQL for some reason doesn't implement efficiently enough.

If it would help anyone to see the query plan or such - I can modify the
logs to show that, just let me know.

Btw - I've let this query run for a while & I haven't seen it complete
... soooo ... I don't know if it would ever complete or not.

Any help at all is as always, appreciated.

Sincerest regards,
--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

After you load the data, you need to run vacuum analzye. That'll
get statistics on the current data in the table. Of course, I'm
not sure that'll help in this case.

On Thu, 8 Feb 2001, Brice Ruth wrote:

What does:

EXPLAIN
SELECT
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.MedCondID,
tblMedCond.PatientName AS MedCondPatientName,
tblMedCond.ProfessionalName AS MedCondProfessionalName,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel
FROM
tblSIDEDrugLink,
tblMedCond
WHERE
(tblSIDEDrugLink.DrugID IN ('DG-18698')) AND
(tblSIDEDrugLink.MedCondID = tblMedCond.MedCondID)
ORDER BY
tblSIDEDrugLink.DrugID,
tblSIDEDrugLink.Frequency,
tblSIDEDrugLink.SeverityLevel,
tblSIDEDrugLink.MedCondID;

Give out? (From the command line psql tool.)

Regards,
Phil

+----------------------------------+
| Phil Mayers, Network Support |
| Centre for Computing Services |
| Imperial College |
+----------------------------------+

-----Original Message-----
From: Brice Ruth [mailto:brice@webprojkt.com]
Sent: 08 February 2001 16:20
To: pgsql-general@postgresql.org.pgsql-sql@postgresql.org
Subject: [GENERAL] Re: Query never returns ...

FYI - I let the query run for 11.5 minutes before killing it off. It
had consumed 11 minutes, 18 seconds of CPU time (reported by ps). The
following messages are from the server log, I'm pasting in all messages
directly following the query:

010208.10:04:29.473 [24041] ProcessQuery
010208.10:15:59.212 [24041] FATAL 1: The system is shutting down
010208.10:15:59.213 [24041] AbortCurrentTransaction

FATAL: s_lock(4001600c) at spin.c:111, stuck spinlock. Aborting.

FATAL: s_lock(4001600c) at spin.c:111, stuck spinlock. Aborting.
Server process (pid 24041) exited with status 134 at Thu Feb 8 10:17:09
2001
Terminating any active server processes...
Server processes were terminated at Thu Feb 8 10:17:09 2001

All right ... after reading up on the documentation for vacuum, I
understand why that's necessary. I've run vacuum analyze on all the
tables, now. Here are the more realistic results from explain:

NOTICE: QUERY PLAN:

Sort (cost=62.46..62.46 rows=14 width=64)
-> Nested Loop (cost=0.00..62.19 rows=14 width=64)
-> Index Scan using pk_tblsidedruglink on tblsidedruglink
(cost=0.00..33.82 rows=14 width=28)
-> Index Scan using pk_tblmedcond on tblmedcond
(cost=0.00..2.01 rows=1 width=36)

The query runs lightning fast now ... THANK YOU!!! :)

-Brice

Stephan Szabo wrote:

After you load the data, you need to run vacuum analzye. That'll
get statistics on the current data in the table. Of course, I'm
not sure that'll help in this case.

On Thu, 8 Feb 2001, Brice Ruth wrote:

Stephan,

Here is what EXPLAIN shows:

NOTICE: QUERY PLAN:

Sort (cost=0.02..0.02 rows=1 width=64)
-> Nested Loop (cost=0.00..0.01 rows=1 width=64)
-> Seq Scan on tblmedcond (cost=0.00..0.00 rows=1 width=36)
-> Seq Scan on tblsidedruglink (cost=0.00..0.00 rows=1 width=28)

As for vacuum analyze - prior to running into these problems, I deleted
all data from the database (using delete from <tblname>) and then ran
vacuumdb -a, after which I loaded the data into the tables using 'copy
... from' - there have been no updates to the database since then -
merely selects.

--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

Brice,

This sounds like a problem with you postgresql install or your HDD
rather than a problem with your query ... which appears to be fine.

I would reccomend taking it up with PGSQL Inc. (or Great Bridge)
pay-for support if this is a commercial project.

-Josh Berkus
--
______AGLIO DATABASE SOLUTIONS___________________________
Josh Berkus
Complete information technology josh@agliodbs.com
and data management solutions (415) 565-7293
for law firms, small businesses fax 621-2533
and non-profit organizations. San Francisco

Brice -
What does EXPLAIN say for that query? With empty tables, I get two index scans,
a merge join, and two sorts. I'm guessing wildly that you've got a non-optimal
sort strategy happening somehow, given the four fold ORDER BY clause.

Ross

Here's the empty version:

NOTICE: QUERY PLAN:

Sort (cost=84.25..84.25 rows=100 width=64)
-> Merge Join (cost=8.30..80.93 rows=100 width=64)
-> Index Scan using pk_tblmedcond on tblmedcond (cost=0.00..60.00 rows=1000 width=36)
-> Sort (cost=8.30..8.30 rows=10 width=28)
-> Index Scan using pk_tblsidedruglink on tblsidedruglink (cost=0.00..8.14 rows=10 width=28)

EXPLAIN

Ross,

Thanx to Stephan's help, I found out that after loading the tables w/
data, I had to run vacuum analyze to inform the optimizer of the amount
of data in the table (amongst other things, I imagine). After running
that on all the tables, the query performs fine.

-Brice

"Ross J. Reedstrom" wrote:

Brice -
What does EXPLAIN say for that query? With empty tables, I get two index scans,
a merge join, and two sorts. I'm guessing wildly that you've got a non-optimal
sort strategy happening somehow, given the four fold ORDER BY clause.

Ross

Here's the empty version:

NOTICE: QUERY PLAN:

Sort (cost=84.25..84.25 rows=100 width=64)
-> Merge Join (cost=8.30..80.93 rows=100 width=64)
-> Index Scan using pk_tblmedcond on tblmedcond (cost=0.00..60.00 rows=1000 width=36)
-> Sort (cost=8.30..8.30 rows=10 width=28)
-> Index Scan using pk_tblsidedruglink on tblsidedruglink (cost=0.00..8.14 rows=10 width=28)

EXPLAIN

On Thu, Feb 08, 2001 at 10:19:43AM -0600, Brice Ruth wrote:

FYI - I let the query run for 11.5 minutes before killing it off. It
had consumed 11 minutes, 18 seconds of CPU time (reported by ps). The
following messages are from the server log, I'm pasting in all messages
directly following the query:

--
Brice Ruth
WebProjkt, Inc.
VP, Director of Internet Technology
http://www.webprojkt.com/

Um, no.

You should run vacuum analyze AFTER you loaded up the data, otherwise,
your table statistics will be all wrong (it'll contain 'empty table'
statistics).

-alex
On Thu, 8 Feb 2001, Brice Ruth wrote:

Brice Ruth <brice@webprojkt.com> writes:

As for vacuum analyze - prior to running into these problems, I deleted
all data from the database (using delete from <tblname>) and then ran
vacuumdb -a, after which I loaded the data into the tables using 'copy
... from' - there have been no updates to the database since then -
merely selects.

That was the wrong order to do things in :-(. The VACUUM ANALYZE posted
statistics showing all your tables as empty, and the planner is now
faithfully choosing plans that are good for tiny tables --- ie, minimal
startup cost and don't worry about per-tuple cost.

There has been some talk of having stats automatically updated by COPY,
but right now it doesn't happen. So the correct procedure is to do a
VACUUM ANALYZE on a table *after* you do any sizable data additions.

BTW, people have occasionally stated on the list that you need to redo
VACUUM ANALYZE after adding/dropping indexes, but that's not true, at
least not in the present state of the world. VACUUM ANALYZE only posts
stats about the data in the table(s). The planner always looks at the
current set of indices for a table, together with the last-posted data
statistics, to choose a plan.

regards, tom lane