PQstatus() detect change in connection...

Started by Matthew Hagertyabout 24 years ago8 messages
#1Matthew Hagerty
mhagerty@voyager.net

Greetings,

PostgreSQL 7.1.3, FreeBSD-4.3-RELEASE, gcc 2.95.3

I'm trying to attempt to detect a failed backend connection, but a call to
PQstatus() always returns the state of the backend when the call was
made. For example, take this test code:

PGconn *pgConn;
PGresult *pgRes;
int fdPGconn;

int i = 0;
int iNewState = 0;
int iOldState = 60;

pgConn = PQconnectdb("dbname=pglogd user=postgres");

while ( i == 0 )
{
iNewState = PQstatus(pgConn);

if ( iNewState != iOldState )
{
iOldState = iNewState;
printf("Connection State [%d]\n", iNewState);

fdPGconn = PQsocket(pgConn);
printf("Connection Socket [%d]\n", fdPGconn);
}

sleep(1);
}

PQfinish(pgConn);

If you start this with the backend running, the status is CONNECTION_OK,
then pull the plug on the backend, the call to PQstatus() will still return
CONNECTION_OK, even though the backend is not running. Start this program
with the backend not running, then start the backend, PQstatus() never sees
the backend come to life...

Am I reading PQstatus() wrong? Is there any way to detect when the backend
goes down or comes back up?

Thanks,
Matthew

#2Mark Pritchard
mark@tangent.net.au
In reply to: Matthew Hagerty (#1)
2 attachment(s)
Re: PQstatus() detect change in connection...

I presume you are trying to re-establish a connection automatically...if
that doesn't apply, ignore the rest of this email :)

The way I interpreted the docs was that you can use the return codes from
PQexec() to establish whether the command was sent to the backend correctly.
PQresultStatus() returns whether the command was syntactically
correct/executed OK.

I've attached a chunk of code from a back-end independent DB driver
(supports Oracle, PgSQL, MySQL through the same front end API), which
implements this auto-reconnect. Take a look at the sqlExec() method.

This code successfully recovers when used in a client connection pool in the
following sequence:

1) start postmaster
2) connect through pool/driver
3) issue SQL statements
4) kill postmaster
5) start postmaster
6) issue SQL statements
7) driver detects connection invalid, reconnects and re-issues
automatically.

Perhaps those infinitely more knowledgeable on the list have a better/more
correct way of doing things?

Cheers,

Mark Pritchard

Show quoted text

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Matthew Hagerty
Sent: Thursday, 18 October 2001 10:47 AM
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] PQstatus() detect change in connection...

Greetings,

PostgreSQL 7.1.3, FreeBSD-4.3-RELEASE, gcc 2.95.3

I'm trying to attempt to detect a failed backend connection, but
a call to
PQstatus() always returns the state of the backend when the call was
made. For example, take this test code:

PGconn *pgConn;
PGresult *pgRes;
int fdPGconn;

int i = 0;
int iNewState = 0;
int iOldState = 60;

pgConn = PQconnectdb("dbname=pglogd user=postgres");

while ( i == 0 )
{
iNewState = PQstatus(pgConn);

if ( iNewState != iOldState )
{
iOldState = iNewState;
printf("Connection State [%d]\n", iNewState);

fdPGconn = PQsocket(pgConn);
printf("Connection Socket [%d]\n", fdPGconn);
}

sleep(1);
}

PQfinish(pgConn);

If you start this with the backend running, the status is CONNECTION_OK,
then pull the plug on the backend, the call to PQstatus() will
still return
CONNECTION_OK, even though the backend is not running. Start
this program
with the backend not running, then start the backend, PQstatus()
never sees
the backend come to life...

Am I reading PQstatus() wrong? Is there any way to detect when
the backend
goes down or comes back up?

Thanks,
Matthew

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Attachments:

DBDriverPostgres.cppapplication/octet-stream; name=DBDriverPostgres.cppDownload
/***************************************************************************

    DBDriverPostgres.cpp

    Date        Author  Description
    20001114    MP      Created
    20010529    OB      Mods to make this TRULY part of TSAUtils

 ***************************************************************************/

#include "DBDriverPostgres.h"

TSAUTILS_NAMESPACE_BEGIN

/*
 ***** Internal function declarations
 */

    // Trims trailing spaces from string
    char * _rTrim(char *in_string);

/*
 ***** Internal function definitions
 */

// _rTrim()
char * _rTrim(char *in_string)
{
    long lPos;

    // Exit if null
    if (!in_string)
        return "";

    // Find last char in string so we can loop backward
    lPos = strlen(in_string)-1;

    // Loop backward finding last non-space char
    while ((lPos >= 0) && (in_string[lPos] == ' '))
        lPos--;

    // Move forward one place (back on to the spaces) and set null
    lPos++;
    in_string[lPos] = 0;

    return in_string;
}

/*
 ***** DBDriverPostgres
 */

// DBDriverPostgres::_initResultSet()
void DBDriverPostgres::_initResultSet()
{
    // Clear the column names
    m_columnNames.clear();

    // Free the result set
    PQclear(m_resultSet);
    m_resultSet        = NULL;

    // Indicate we did not return rows
    m_blDidReturnRows  = false;

    // Reset the row number
    m_lCurrentRow      = -1;
}

// DBDriverPostgres::_validateFieldAccess()
int DBDriverPostgres::_validateFieldAccess(const string &in_sFldName)
{
    int iFieldIndex;

    // Check if we have rows
    if (!didReturnRows())
        throw TSAUTILS::xExceptionGeneric("No records returned");

    // Check if the field name is valid and return the index if it is
    iFieldIndex = PQfnumber(m_resultSet,in_sFldName.c_str());
    if (iFieldIndex == -1)
        throw TSAUTILS::xExceptionGeneric("Invalid field name %s",in_sFldName.c_str());

    return iFieldIndex;
}

// DBDriverPostgres::colNames
vector <string> DBDriverPostgres::colNames()
{
    // Check if we have rows
    if (!didReturnRows())
        throw TSAUTILS::xExceptionGeneric("Column names unavailable - query did not return rows");

    // Return the list of column names
    return m_columnNames;
}

// DBDriverPostgres::DBDriverPostgres
DBDriverPostgres::DBDriverPostgres(const char *pchHost, const char *pchDB, const char *pchUser, const char *pchPwd)
{
    // Pre-init
    m_resultSet = NULL;

    // Init
    _initResultSet();

    // Build connect string
    sprintf(m_pchTemp,"host=%s dbname=%s user=%s password=%s",pchHost,pchDB,pchUser,pchPwd);

    // Connect to DB
    m_connection = PQconnectdb(m_pchTemp);
    if (PQstatus(m_connection) == CONNECTION_BAD)
        throw TSAUTILS::xExceptionGeneric("Unable to connect to %s on %s. Error - %s",pchDB,pchHost,PQerrorMessage(m_connection));
}

// DBDriverPostgres::~DBDriverPostgres
DBDriverPostgres::~DBDriverPostgres()
{
    // Free result set
    _initResultSet();

    // Free connection
    PQfinish(m_connection);
}

// DBDriverPostgres::fldBool
bool DBDriverPostgres::fldBool(const string &in_sFldName)
{
    // Return whether int <> 0
    return fldInt(in_sFldName);
}

// DBDriverPostgres::fldDbl
double DBDriverPostgres::fldDbl(const string &in_sFldName)
{
    int iFieldIndex = _validateFieldAccess(in_sFldName);

    // Return field value
    return atof(PQgetvalue(m_resultSet,m_lCurrentRow,iFieldIndex));
}

// DBDriverPostgres::fldDT
Date DBDriverPostgres::fldDT(const string &in_sFldName)
{
    // Grab field index and data
    int iFieldIndex     = _validateFieldAccess(in_sFldName);
    string sDateTime    = PQgetvalue(m_resultSet,m_lCurrentRow,iFieldIndex);

    // Trim off the timezone
    size_t iTZstart = sDateTime.find('+');
    if (iTZstart < sDateTime.length())
        sDateTime = sDateTime.substr(0,iTZstart);

    // Return a new date object
    return Date(sDateTime);
}

// DBDriverPostgres::fldFlt
float DBDriverPostgres::fldFlt(const string &in_sFldName)
{
    int iFieldIndex = _validateFieldAccess(in_sFldName);

    // Return field value
    return atof(PQgetvalue(m_resultSet,m_lCurrentRow,iFieldIndex));
}

// DBDriverPostgres::fldInt
int DBDriverPostgres::fldInt(const string &in_sFldName)
{
    int iFieldIndex = _validateFieldAccess(in_sFldName);

    // Return field value
    return atoi(PQgetvalue(m_resultSet,m_lCurrentRow,iFieldIndex));
}

// DBDriverPostgres::fldLong
long DBDriverPostgres::fldLong(const string &in_sFldName)
{
    int iFieldIndex = _validateFieldAccess(in_sFldName);

    // Return field value
    return atol(PQgetvalue(m_resultSet,m_lCurrentRow,iFieldIndex));
}

// DBDriverPostgres::fldStr
string DBDriverPostgres::fldStr(const string &in_sFldName)
{
	string sRet = "";
    int iFieldIndex = _validateFieldAccess(in_sFldName);

    // Return field value
    char *pchTmp = PQgetvalue(m_resultSet,m_lCurrentRow,iFieldIndex);
    if (pchTmp)
    	sRet = _rTrim(pchTmp);
   
   	return sRet;
}

// DBDriverPostgres::didReturnRows
bool DBDriverPostgres::didReturnRows()
{
    return m_blDidReturnRows;
}

// DBDriverPostgres::recordNext
bool DBDriverPostgres::recordNext()
{
    // Return false if we didn't return any rows from the query or we are at end
    if ((!didReturnRows()) || (m_lCurrentRow >= PQntuples(m_resultSet)))
        return false;

    // Move to the next row
    m_lCurrentRow++;

    return (m_lCurrentRow < PQntuples(m_resultSet));
}

// DBDriverPostgres::sqlExec
bool DBDriverPostgres::sqlExec(const char *in_pchSQL)
{
    ExecStatusType execStatus;
    int iFieldCount;

    // Init
    _initResultSet();

    // Check if we have SQL
    if (!in_pchSQL)
        throw TSAUTILS::xExceptionGeneric("Unable to execute query - no SQL specified");

    // Attempt to execute it
    m_resultSet    = PQexec(m_connection,in_pchSQL);
    if (!m_resultSet)
    {
        // Query execution failed. Attempt to reconnect and re-execute
        PQreset(m_connection);
        m_resultSet = PQexec(m_connection,in_pchSQL);
    }

    // Check the result set status
    execStatus      = PQresultStatus(m_resultSet);
    if ((!m_resultSet) || ((execStatus != PGRES_COMMAND_OK) && (execStatus != PGRES_TUPLES_OK)))
        throw TSAUTILS::xExceptionGeneric("Error executing query [%s]. Error = %s",in_pchSQL,PQerrorMessage(m_connection));

    // Exit if we executed a query that doesn't return rows
    if (execStatus == PGRES_COMMAND_OK)
        return didReturnRows();

    // Exit if we executed a query that returns rows, but didn't in this case (select with no matching records)
    if (PQntuples(m_resultSet) == 0)
        return didReturnRows();

    // We have data
    m_blDidReturnRows = true;

    // Init row counter
    m_lCurrentRow = -1;

    // Fetch the list of column names
    iFieldCount = PQnfields(m_resultSet);
    for (int i = 0; i < iFieldCount; i++)
        m_columnNames.push_back(string(PQfname(m_resultSet,i)));

    // Indicate we returned rows
    return didReturnRows();
}

// DBDriverPostgres::transCommit
void DBDriverPostgres::transCommit()
{
    _initResultSet();

    // Start the transaction
    m_resultSet = PQexec(m_connection, "COMMIT");
    if ((!m_resultSet) || (PQresultStatus(m_resultSet) != PGRES_COMMAND_OK))
        throw TSAUTILS::xExceptionGeneric("Unable to start transaction. Error - %s",PQerrorMessage(m_connection));

    _initResultSet();
}

// DBDriverPostgres::transRollback
void DBDriverPostgres::transRollback()
{
    _initResultSet();

    // Start the transaction
    m_resultSet = PQexec(m_connection, "ROLLBACK");
    if ((!m_resultSet) || (PQresultStatus(m_resultSet) != PGRES_COMMAND_OK))
        throw TSAUTILS::xExceptionGeneric("Unable to rollback transaction. Error - %s",PQerrorMessage(m_connection));

    _initResultSet();
}

// DBDriverPostgres::transStart
void DBDriverPostgres::transStart()
{
    _initResultSet();

    // Start the transaction
    m_resultSet = PQexec(m_connection, "BEGIN");
    if ((!m_resultSet) || (PQresultStatus(m_resultSet) != PGRES_COMMAND_OK))
        throw TSAUTILS::xExceptionGeneric("Unable to start transaction. Error - %s",PQerrorMessage(m_connection));

    _initResultSet();
}

TSAUTILS_NAMESPACE_END
DBDriverPostgres.happlication/octet-stream; name=DBDriverPostgres.hDownload
#ifndef __TSAUTILS_DB_DRIVER_POSTGRES_H__
#define __TSAUTILS_DB_DRIVER_POSTGRES_H__

/***************************************************************************

    DBDriverPostgres.h

    DBDriverPostgres implements Postgres functionality

    Date        Author  Description
    20001114    MP      Created

 ***************************************************************************/

#include <postgres/libpq-fe.h>

#include "DBDriverGeneric.h"

TSAUTILS_NAMESPACE_BEGIN

/*
 ***** DBDriverPostgres
 */

class DBDriverPostgres : public DBDriverGeneric
{
    private:
        // Whether the last query returned rows
        bool    m_blDidReturnRows;

        // Connection to Postgres database
        PGconn *m_connection;

        // Result set from Postgres
        PGresult *m_resultSet;

        // Column names in the result set
        vector <string> m_columnNames;

        // Current row number
        long m_lCurrentRow;

        // Initialises result set data
        void _initResultSet();

        // Validates OK to access fields
        int _validateFieldAccess(const string &in_sFldName);
    public:
        // Constructor/destructor
        DBDriverPostgres(const char *pchHost, const char *pchDB, const char *pchUser, const char *pchPwd);
        virtual ~DBDriverPostgres();

		// Returns column names
		vector <string> colNames();

        // Retrieves field data
        bool        fldBool(const string &in_sFldName);
        double      fldDbl(const string &in_sFldName);
        Date        fldDT(const string &in_sFldName);
        float       fldFlt(const string &in_sFldName);
        int         fldInt(const string &in_sFldName);
        long        fldLong(const string &in_sFldName);
        string      fldStr(const string &in_sFldName);

        // Record navigation
        bool    didReturnRows();
        bool    recordNext();

        // SQL manipulation
        bool    sqlExec(const char *in_pchSQL);

        // Transaction control
        void    transCommit();
        void    transRollback();
        void    transStart();
};

TSAUTILS_NAMESPACE_END

#endif
#3Matthew Hagerty
mhagerty@voyager.net
In reply to: Mark Pritchard (#2)
Re: PQstatus() detect change in connection...

I am trying to re-establish a connection, however, I cannot afford to issue
a query to determine if the connection still exists. I'm writing a server
that uses the asynchronous query processing functions and speed is an
issue. Queries are slow compared to what the server does and it cannot
wait around for a query to finish just to see if another query *should* be
attempted based on the connection status.

I've been digging into the libpq code to see what is going on, maybe I can
gleam a little hint or two there... Anyone know a good *fast* way to test
if a socket is still valid?

Thanks,
Matthew

At 11:51 AM 10/18/2001 +1000, Mark Pritchard wrote:

Show quoted text

I presume you are trying to re-establish a connection automatically...if
that doesn't apply, ignore the rest of this email :)

The way I interpreted the docs was that you can use the return codes from
PQexec() to establish whether the command was sent to the backend correctly.
PQresultStatus() returns whether the command was syntactically
correct/executed OK.

I've attached a chunk of code from a back-end independent DB driver
(supports Oracle, PgSQL, MySQL through the same front end API), which
implements this auto-reconnect. Take a look at the sqlExec() method.

This code successfully recovers when used in a client connection pool in the
following sequence:

1) start postmaster
2) connect through pool/driver
3) issue SQL statements
4) kill postmaster
5) start postmaster
6) issue SQL statements
7) driver detects connection invalid, reconnects and re-issues
automatically.

Perhaps those infinitely more knowledgeable on the list have a better/more
correct way of doing things?

Cheers,

Mark Pritchard

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Matthew Hagerty
Sent: Thursday, 18 October 2001 10:47 AM
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] PQstatus() detect change in connection...

Greetings,

PostgreSQL 7.1.3, FreeBSD-4.3-RELEASE, gcc 2.95.3

I'm trying to attempt to detect a failed backend connection, but
a call to
PQstatus() always returns the state of the backend when the call was
made. For example, take this test code:

PGconn *pgConn;
PGresult *pgRes;
int fdPGconn;

int i = 0;
int iNewState = 0;
int iOldState = 60;

pgConn = PQconnectdb("dbname=pglogd user=postgres");

while ( i == 0 )
{
iNewState = PQstatus(pgConn);

if ( iNewState != iOldState )
{
iOldState = iNewState;
printf("Connection State [%d]\n", iNewState);

fdPGconn = PQsocket(pgConn);
printf("Connection Socket [%d]\n", fdPGconn);
}

sleep(1);
}

PQfinish(pgConn);

If you start this with the backend running, the status is CONNECTION_OK,
then pull the plug on the backend, the call to PQstatus() will
still return
CONNECTION_OK, even though the backend is not running. Start
this program
with the backend not running, then start the backend, PQstatus()
never sees
the backend come to life...

Am I reading PQstatus() wrong? Is there any way to detect when
the backend
goes down or comes back up?

Thanks,
Matthew

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matthew Hagerty (#3)
Re: PQstatus() detect change in connection...

Matthew Hagerty <mhagerty@voyager.net> writes:

Anyone know a good *fast* way to test
if a socket is still valid?

What exactly are you trying to defend against?

In general, I don't believe that there is any way of discovering whether
the server is still up, other than to send it a query. (FWIW, an empty
query string bounces back very quickly, with little processing.)

For particular scenarios it's possible that some notification has been
delivered to the client, but if you have had (say) a loss of network
connectivity then there just is no other alternative. Your end isn't
going to discover the connectivity loss until it tries to send a
message.

regards, tom lane

#5Matthew Hagerty
mhagerty@voyager.net
In reply to: Tom Lane (#4)
Re: PQstatus() detect change in connection...

At 02:10 PM 10/18/2001 -0400, Tom Lane wrote:

Matthew Hagerty <mhagerty@voyager.net> writes:

Anyone know a good *fast* way to test
if a socket is still valid?

What exactly are you trying to defend against?

In general, I don't believe that there is any way of discovering whether
the server is still up, other than to send it a query. (FWIW, an empty
query string bounces back very quickly, with little processing.)

For particular scenarios it's possible that some notification has been
delivered to the client, but if you have had (say) a loss of network
connectivity then there just is no other alternative. Your end isn't
going to discover the connectivity loss until it tries to send a
message.

regards, tom lane

I was using PQstatus() under the assumption that it actually *checked* the
connection, however I have since discovered that is simply returns the
value in a structure, and that value only gets updated in pqReadData() or
pqReadReady() (both of which are internal function calls.)

What I'm doing is using the asynchronous processing to write a server that
does not have to wait around for a query to finish (which is a slow process
compared to what the rest of the server does.) So, using a query to test
if the connection is up seems rather redundant and slow... I was hoping to
come up with a faster more simple solution. If the connection is down I
need to write - what would have been a query - to a temporary place and
attempt a reconnect, all while going off and doing other things.

This all came about when my main select() bailed because the backend went
down and the socket's file-descriptor became invalid. I could probably
catch the error in that loop, but I also want to check the connection
*before* submitting a query... Basically, I hope to avoid a huge rewrite
based on my assumption of how PQstatus() was actually working. ;-)

Currently I'm looking at fnctl() or a dedicated select() call (similar to
what pgReadReady() does), but I'm not sure of the OS overhead of these
solutions compared to each other or an empty query. Any insight would be
greatly appreciated.

Thanks,
Matthew

#6Peter Eisentraut
peter_e@gmx.net
In reply to: Matthew Hagerty (#3)
Re: PQstatus() detect change in connection...

Matthew Hagerty writes:

I am trying to re-establish a connection, however, I cannot afford to issue
a query to determine if the connection still exists.

But requesting that the server do something *is* the only way to know
whether it's still alive. Another question to ask, of course, would be,
why is your server always going down?

I've been digging into the libpq code to see what is going on, maybe I can
gleam a little hint or two there... Anyone know a good *fast* way to test
if a socket is still valid?

Try to send or receive something.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

#7Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matthew Hagerty (#5)
Re: PQstatus() detect change in connection...

Matthew Hagerty <mhagerty@voyager.net> writes:

but I also want to check the connection
*before* submitting a query...

This strikes me as utterly pointless. You'll need to be able to recover
from query failure anyway, so what's the value of testing beforehand?
Send the query and see if it works or not.

regards, tom lane

#8Zeugswetter Andreas SB SD
ZeugswetterA@spardat.at
In reply to: Tom Lane (#7)
Re: PQstatus() detect change in connection...

Matthew Hagerty <mhagerty@voyager.net> writes:

but I also want to check the connection *before* submitting a

query...

If you mean directly before the query, then forget it, as Tom already
said :-)

This strikes me as utterly pointless. You'll need to be able to

recover

from query failure anyway, so what's the value of testing beforehand?
Send the query and see if it works or not.

I see a value in checking connection status before you start doing
loads of local work after a long idle time, that results in a query.
In this situation I guess it is good enough to send an empty query
even if it takes a little.

In our projects we recv 0 bytes from the socket every x seconds
during long idle periods to detect connection problems early.
While it is not 100% reliable (since it does not transfer
anything over the network) it does detect some common error situations.

I am not 100% sure, but I think PQstatus could be patched to do that.

Andreas