serverless postgresql

* Jeff Bowden <jlb@houseofdistraction.com> [2004-01-13 13:38:02 -0800]:

For ease of configuration and other reasons, I would like for my
single-user GUI app to be able to use postgresql in-process as a library
accessing a database created in the users home directory. I think I
could possibly get what I want by launching a captive copy of postmaster
with appropriate args but it seems conceptually cleaner to not have a
seperate process at all. Has anyone tried to do anything like this?

hmm, i've also played around a little bit w/ postgresql standalone.
you could also do nice things like calling postmaster from inetd ;-)

but I dont think you really wanna have the postmaster in the same
process w/ your application, since it relies on forks, signals,
mmap() and some other things which directly affect the process control -
this probably infers with your application. On the application side
you cant really control, what it does, so it sounds quite difficult.
Having the postmaster in a separate process connected by a socket or
pty seems to be the better solution.

cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT services

phone: +49 36207 519931 www: http://www.metux.de/
fax: +49 36207 519932 email: contact@metux.de
cellphone: +49 174 7066481
---------------------------------------------------------------------
Diese Mail wurde mit UUCP versandt. http://www.metux.de/uucp/

I have just about the same sort of needs now and concluded that postgres
just is not suited for embedding into apps like that. I am going with
sqlite and it is working fairly well. We just made user defined functions
in php for sqlite to match the date functions in postgres. Who knows what
other issues we may run into but so far sqlite is working fairly well.

I think that sqlite stores everything as a string but for searching,
sorting, ordering etc it uses the type info.

I to would absolutely love to just include a postgres dll and have postgres
in process and just store everything in a nice little file but from what I
have read hear that would involve major changes that the developers on not
interested in making. So far sqlite has done the job for me.

Also I am considering looking into firebird for an embedded database
solution since it can you pretty much what you are talking about here with
an in proccess db that uses a single file to store it's data. While I don't
want to switch my server stuff to it, it is probably more full featured than
sqlite (I'm sure it has date and arithmatic functions) and is made to work
well in embedded situations.

I would be intersted to see what you end up doing.

Rick

----- Original Message -----
From: "Jeff Bowden" <jlb@houseofdistraction.com>
To: "pgsql-general" <pgsql-general@postgresql.org>
Sent: Tuesday, January 13, 2004 2:38 PM
Subject: [GENERAL] serverless postgresql

"Rick Gigger" <rick@alpinenetworking.com> writes:

I to would absolutely love to just include a postgres dll and have postgres
in process and just store everything in a nice little file but from what I
have read hear that would involve major changes that the developers on not
interested in making.

Not only are the developers uninterested in it, the developers actively
oppose it. We think an embedded database library cannot be reliable
enough to meet our notion of a "database", since it would be subject to
failures anytime the surrounding application has a bug. Keeping the
client code in a separate process is a far more robust design.

regards, tom lane

Tom Lane wrote:

"Rick Gigger" <rick@alpinenetworking.com> writes:

I to would absolutely love to just include a postgres dll and have postgres
in process and just store everything in a nice little file but from what I
have read hear that would involve major changes that the developers on not
interested in making.

Not only are the developers uninterested in it, the developers actively
oppose it. We think an embedded database library cannot be reliable
enough to meet our notion of a "database", since it would be subject to
failures anytime the surrounding application has a bug. Keeping the
client code in a separate process is a far more robust design.

That makes sense to me. I wonder if sqlite suffers for this problem
(e.g. app crashing and corrupting the database).

What about the notion of running postmaster on-demand as the user? Is
that something that anyone has experience with? It seems like it would
solve the complex configuration problems without compromising robustness
or requiring any special support other than sufficient command-line
parameters.

Oh yeah, that brings me to another question. I was looking at the
postmaster command-line switches and I couldn't find any that would
allow me to point it at an arbitrary config file but then I had a look
around and it seems that the .conf files are already used by the various
startup scripts. Does postmaster itself ever read the .conf files or is
it controlled strictly by switches?

On Jan 13, 2004, at 13:38, Jeff Bowden wrote:

but it seems conceptually cleaner to not have a seperate process at
all.

That depends a lot on what you mean by cleaner. A few small programs
in their own process space dedicated to a specific set of functionality
with well-defined interfaces can make for a much more reliable
application (see postfix).

--
Dustin Sallings

Jeff Bowden <jlb@houseofdistraction.com> writes:

That makes sense to me. I wonder if sqlite suffers for this problem
(e.g. app crashing and corrupting the database).

Likely. I can tell you that Ann Harrison once told me she made a decent
amount of money as a consultant fixing broken Interbase/Firebird
database files. It would be hard to make a living in the same game for
Postgres. Now I don't think that Firebird is any buggier than Postgres.
But it comes in an embedded-library form; I'll bet lunch that most of
those data corruption problems were actually induced by crashes of
surrounding applications.

What about the notion of running postmaster on-demand as the user?

Possibly. You'd have to think carefully about what conditions the
postmaster should be shut down under, and especially what conditions
it should NOT be shut down under --- eg, a kill to the parent client
application shouldn't cause an ungraceful postmaster exit. It could
be tricky to get the signal handling right, especially under shells that
try to deliver signals to all children of a process being signaled.
On the whole I suspect it'd be easier just to leave the postmaster
running in the background...

Oh yeah, that brings me to another question. I was looking at the
postmaster command-line switches and I couldn't find any that would
allow me to point it at an arbitrary config file

The config files all live in $PGDATA and so are determined by the -D
switch. There was some talk of changing this, awhile back, but it
foundered on lack of consensus about exactly what to do instead.

regards, tom lane

jlb@houseofdistraction.com (Jeff Bowden) wrote:

Tom Lane wrote:

Not only are the developers uninterested in it, the developers actively
oppose it. We think an embedded database library cannot be reliable
enough to meet our notion of a "database", since it would be subject to
failures anytime the surrounding application has a bug. Keeping the
client code in a separate process is a far more robust design.

That makes sense to me. I wonder if sqlite suffers for this problem
(e.g. app crashing and corrupting the database).

I'm not sure its use has yet been so widespread that there is a good
feel for this. It is doubtless _possible_; "crash patterns" likely
_don't_ lead to enormous disasters when databases are small,
localized, and the apps probably _don't_ crash all the time.

What about the notion of running postmaster on-demand as the user?
Is that something that anyone has experience with? It seems like it
would solve the complex configuration problems without compromising
robustness or requiring any special support other than sufficient
command-line parameters.

I haven't had call for this being controlled "by the application," but
I can't see it being vastly troublesome. And I _don't_ see it
requiring lots of "command line" parameters; you just need to specify
the directory where the configuration is.

Oh yeah, that brings me to another question. I was looking at the
postmaster command-line switches and I couldn't find any that would
allow me to point it at an arbitrary config file but then I had a
look around and it seems that the .conf files are already used by
the various startup scripts. Does postmaster itself ever read the
.conf files or is it controlled strictly by switches?

All of the .conf files are in one directory, and that directory is
controlled by either the value of environment variable PGDATA or the
"-D" command parameter.

It seems not-overly-valuable to have the .conf files be able to be
specified in random other locations.

Is there some particular reason you have in mind why you would want to
_ignore_ the configuration in $HOME/DBDIR and instead use
configuration in some other random location? I would think that
collecting the config into one directory, as is done right now, is a
_good_ thing.
--
let name="cbbrowne" and tld="ntlug.org" in name ^ "@" ^ tld;;
http://www3.sympatico.ca/cbbrowne/lsf.html
"Even in the area of anticompetitive conduct, Microsoft is mainly an
imitator." -- Ralph Nader (1998/11/11)

Tom Lane wrote:

Jeff Bowden <jlb@houseofdistraction.com> writes:

What about the notion of running postmaster on-demand as the user?

Possibly. You'd have to think carefully about what conditions the
postmaster should be shut down under, and especially what conditions
it should NOT be shut down under --- eg, a kill to the parent client
application shouldn't cause an ungraceful postmaster exit. It could
be tricky to get the signal handling right, especially under shells that
try to deliver signals to all children of a process being signaled.
On the whole I suspect it'd be easier just to leave the postmaster
running in the background...

Details, details.... :-)

Oh yeah, that brings me to another question. I was looking at the
postmaster command-line switches and I couldn't find any that would
allow me to point it at an arbitrary config file

The config files all live in $PGDATA and so are determined by the -D
switch. There was some talk of changing this, awhile back, but it
foundered on lack of consensus about exactly what to do instead.

As long as it can be done.

Quoth jlb@houseofdistraction.com (Jeff Bowden):

For ease of configuration and other reasons, I would like for my
single-user GUI app to be able to use postgresql in-process as a
library accessing a database created in the users home directory. I
think I could possibly get what I want by launching a captive copy of
postmaster with appropriate args but it seems conceptually cleaner to
not have a seperate process at all. Has anyone tried to do anything
like this?

One of the properties of Unix is that spawning extra processes to "do
their thing" allows each application to be designed more 'cleanly.'

Curiously enough, the implementors of alternatives to the Sendmail MTA
have typically concluded that it was not only conceptually cleaner to
have a set of separate cooperating processes, but that they would also
reap benefits in terms of speed and improved security.

Thus, I think you'll find that people disagree with you on this. It
seems to me that it is conceptually cleaner to have the "application"
do "application stuff," and not muddle things up by mixing that with
the "database stuff."
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne&gt; rate me
http://cbbrowne.com/info/sgml.html
Rules of the Evil Overlord #87. "My vats of hazardous chemicals will
be covered when not in use. Also, I will not construct walkways above
them." <http://www.eviloverlord.com/&gt;

Tom Lane <tgl@sss.pgh.pa.us> said:

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Re: tip 9 :-). In 7.3.5 is it possible that the planner is finding a type
mismatch when there isn't one, resulting in a seq scan being selected? I've
had a problem with joining tables on an int8 column (primary key column in one
of the tables). Tested left outer and where clause syntax.

TIA,

Jim Wilson

"Jim Wilson" <jimw@kelcomaine.com> writes:

Re: tip 9 :-). In 7.3.5 is it possible that the planner is finding a type
mismatch when there isn't one, resulting in a seq scan being selected?

When you gave no details, how is anyone to know? For questions like
this, you need to post the table definitions, the exact query, and the
results of EXPLAIN ANALYZE on that query. BTW, pgsql-performance is a
more appropriate list than -general for such questions.

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> said:

"Jim Wilson" <jimw@kelcomaine.com> writes:

Re: tip 9 :-). In 7.3.5 is it possible that the planner is finding a type
mismatch when there isn't one, resulting in a seq scan being selected?

When you gave no details, how is anyone to know? For questions like
this, you need to post the table definitions, the exact query, and the
results of EXPLAIN ANALYZE on that query. BTW, pgsql-performance is a
more appropriate list than -general for such questions.

regards, tom lane

My apologies. I think you did answer my question though, that there isn't a
well known problem as described (in it's general scope). As soon as I get a
chance I will put together a test case and post the issue to psql-performance.

Best,

Jim Wilson

Jeff Bowden <jlb@houseofdistraction.com> writes:

That makes sense to me. I wonder if sqlite suffers for this problem
(e.g. app crashing and corrupting the database).

Likely. I can tell you that Ann Harrison once told me she made a decent
amount of money as a consultant fixing broken Interbase/Firebird
database files. It would be hard to make a living in the same game for
Postgres. Now I don't think that Firebird is any buggier than Postgres.
But it comes in an embedded-library form; I'll bet lunch that most of
those data corruption problems were actually induced by crashes of
surrounding applications.

I remember reading about how some the original commerical code from
Interbase came with some bugs that corrupted databases and that those bugs
were eventually fixed in Firebird. Don't know if that's related to this but
it could be.

Anyway since postgres uses WAL files to verify the integrity of the database
couldn't it more or less make the same guarantee's in an embedded version?
As long as the app uses the db libs unmodified and doesn't mess with the
files it creates how does simply making it embedded increase the change of
db errors resulting in database corruption?

In addition there are times when you want the ease of a relational database
but you don't need say the same level of reliability that you would on your
server. In these cases it would be nice to just have something that is easy
to drop into your app and have it just work. That being said I think that
in these cases the developer isn't usually going to care if it runs in
process or not. Maybe a good solution would be to make a postgres library
that could be included in process that would interface with the application.
All it would do is manager spawning the real postgres process. Then you
would have the reliability of a real server but the ease of use of an
embedded database. Couldn't that end up being pretty transparent to the
developer? If it was made to be easy postgres could become very popular as
an "embedded" db solution but still not have it's reputation diluted by
rampant database corruption from the application.

"Rick Gigger" <rick@alpinenetworking.com> writes:

Anyway since postgres uses WAL files to verify the integrity of the database
couldn't it more or less make the same guarantee's in an embedded version?
As long as the app uses the db libs unmodified and doesn't mess with the
files it creates how does simply making it embedded increase the change of
db errors resulting in database corruption?

If an app's internal data structures get corrupted and it goes nuts,
pretty much anything can happen to its address space.

-Doug

Anyway since postgres uses WAL files to verify the integrity of the database
couldn't it more or less make the same guarantee's in an embedded version?
As long as the app uses the db libs unmodified and doesn't mess with the
files it creates how does simply making it embedded increase the change of
db errors resulting in database corruption?

I have a different idea. I've been thinking about coding it, but haven't
had the time. This could be done with no changes to Postgres itself.

Basically, you would have a library which exported functions such as

pg_instance *pg_start(char *directory);
pg_get_connection(pg_instance *);
pg_stop(pg_instance *);
pg_initdb(char *directory)

pg_start would do the following:
1) check "directory" for an instance of the UNIX socket.
if it is there, make a Postgres connection
if it is not there, start the Postmaster server with "-k directory
-D directory/data" and then make a Postgres connection
2) Create a struct to contain the directory and any other data item we
need to connect to the database
3) Return this structure

pg_get_connection would just be a wrapper for pq_connect()

pg_stop would kill the database.

pg_initdb would simply run initdb

Does anyone see a reason why this wouldn't work?

Jon

This is just what I recommended (I think) in my earlier post. This would be
FANTASTIC. Then I could scrap sqlite and any sqlite custom code that I end
up having to write (there is a little). Plus I could have the reliability
of a multi-process postgres "embedded".

Now if they just finish the Win32 port sometime soon...

----- Original Message -----
From: "Jonathan Bartlett" <johnnyb@eskimo.com>
To: "Rick Gigger" <rick@alpinenetworking.com>
Cc: "Jeff Bowden" <jlb@houseofdistraction.com>; "Tom Lane"
<tgl@sss.pgh.pa.us>; <pgsql-general@postgresql.org>
Sent: Wednesday, January 14, 2004 2:07 PM
Subject: Re: [GENERAL] serverless postgresql

Anyway since postgres uses WAL files to verify the integrity of the

database

couldn't it more or less make the same guarantee's in an embedded

version?

As long as the app uses the db libs unmodified and doesn't mess with the
files it creates how does simply making it embedded increase the change

of

Jeff Bowden wrote:

For ease of configuration and other reasons, I would like for my
single-user GUI app to be able to use postgresql in-process as a library
accessing a database created in the users home directory. I think I
could possibly get what I want by launching a captive copy of postmaster
with appropriate args but it seems conceptually cleaner to not have a
seperate process at all. Has anyone tried to do anything like this?

[Sorry for not actually answering this question]

I believe the demands for embedded/"serverless" version of PostgreSQL to
increase significantly once PostgreSQL is natively available on Windows.
So I would expect that official embedded support to follow quite shortly
after win32 port has stabilized. :-)

--
dave

Rick Gigger wrote:

I have just about the same sort of needs now and concluded that postgres
just is not suited for embedding into apps like that.

Why not? It's not that the PostgreSQL backend is a mammoth like Oracle.
The Firebird embedded version is pretty much the same as their server,
but without network and client authentication layer.

However, embedded usually demands that the backend be threaded.
Otherwise it will be pretty useless/very inconvenient to use in many
apps. Perhaps this is the major change that's hard to do?

--
dave

Tom Lane wrote:

Jeff Bowden <jlb@houseofdistraction.com> writes:

That makes sense to me. I wonder if sqlite suffers for this problem
(e.g. app crashing and corrupting the database).

Likely. I can tell you that Ann Harrison once told me she made a decent
amount of money as a consultant fixing broken Interbase/Firebird
database files. It would be hard to make a living in the same game for
Postgres. Now I don't think that Firebird is any buggier than Postgres.
But it comes in an embedded-library form; I'll bet lunch that most of
those data corruption problems were actually induced by crashes of
surrounding applications.

Do the developers generally oppose the idea of a threaded (but
non-embedded) backend as well? If the backend is thread-safe, then users
can still choose to run multiprocess or multithreaded right?

--
dave