Please advise features in 7.1

"John Huttley" <John@mwk.co.nz> writes:

Reason: I want to know if any of these features are scheduled.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

You can do that now (and for many versions past) with a trigger.
It's not quite as convenient as it ought to be, but it's possible.
AFAIK there's no change in that situation for 7.1.

2. Any parameterised triggers

We've had parameterized triggers for years. Maybe you attach some
meaning to that term beyond what I do?

3. Any parameterised stored procedures that return a result set.

There is some support (dating back to Berkeley Postquel) for functions
returning sets, but it's pretty ugly and limited. Proper support might
happen in 7.2 ...

regards, tom lane

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "John Huttley" <John@mwk.co.nz>
Cc: <pgsql-hackers@postgresql.org>
Sent: Thursday, 23 November 2000 19:05
Subject: Re: [HACKERS] Please advise features in 7.1

"John Huttley" <John@mwk.co.nz> writes:

Reason: I want to know if any of these features are scheduled.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

You can do that now (and for many versions past) with a trigger.
It's not quite as convenient as it ought to be, but it's possible.
AFAIK there's no change in that situation for 7.1.

Yes, Perhaps defining the table with a dummy field and setting up a
'before'
trigger which replaced that field with a calculated value?

Messy but feasible.

2. Any parameterised triggers

We've had parameterized triggers for years. Maybe you attach some
meaning to that term beyond what I do?

I'm referring to the manual that says functions used for triggers must have
no parameters
and return a type Opaque. And indeed it is impossible to create a trigger
from a plSQL function that takes any parameters.

Thus if we have a lot of triggers which are very similar, we cannot just use
one function
and pass an identifying parameter or two to it. We must create an
individual function for each trigger.

Its irritating more than fatal.

3. Any parameterised stored procedures that return a result set.

There is some support (dating back to Berkeley Postquel) for functions
returning sets, but it's pretty ugly and limited. Proper support might
happen in 7.2 ...

Something to look forward to! Meanwhile I'll have a play and see if its
possible to use a read trigger
to populate a temporary table. hmm, that might require a statement level
trigger. Another thing for 7.2,
i guess.

The application programming we are doing now utilises stored procedures
returning record sets
(MSSQL) and the lack is showstopper in our migration plans. Sigh.

Thanks Tom

Regards

John

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "John Huttley" <John@mwk.co.nz>
Cc: <pgsql-hackers@postgresql.org>
Sent: Thursday, 23 November 2000 19:05
Subject: Re: [HACKERS] Please advise features in 7.1

"John Huttley" <John@mwk.co.nz> writes:

Reason: I want to know if any of these features are scheduled.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

You can do that now (and for many versions past) with a trigger.
It's not quite as convenient as it ought to be, but it's possible.
AFAIK there's no change in that situation for 7.1.

Yes, Perhaps defining the table with a dummy field and setting up a
'before'
trigger which replaced that field with a calculated value?

Messy but feasible.

2. Any parameterised triggers

We've had parameterized triggers for years. Maybe you attach some
meaning to that term beyond what I do?

I'm referring to the manual that says functions used for triggers must have
no parameters
and return a type Opaque. And indeed it is impossible to create a trigger
from a plSQL function that takes any parameters.

Thus if we have a lot of triggers which are very similar, we cannot just use
one function
and pass an identifying parameter or two to it. We must create an
individual function for each trigger.

Its irritating more than fatal.

3. Any parameterised stored procedures that return a result set.

There is some support (dating back to Berkeley Postquel) for functions
returning sets, but it's pretty ugly and limited. Proper support might
happen in 7.2 ...

Something to look forward to! Meanwhile I'll have a play and see if its
possible to use a read trigger
to populate a temporary table. hmm, that might require a statement level
trigger. Another thing for 7.2,
i guess.

The application programming we are doing now utilises stored procedures
returning record sets
(MSSQL) and the lack is showstopper in our migration plans. Sigh.

Thanks Tom

Regards

John

Reason: I want to know if any of these features are scheduled.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

This is currently easily done with a procedure that takes a tabletype parameter
with the name the_sum returning the sum of a + b.

Create table test (
A Integer,
B integer
);

create function the_sum (test) returns integer as
'
begin;
return ($1.a + $1.b);
end;
' language 'plpgsql';

A select * won't return the_sum, but a
select t.a, t.b, t.the_sum from test t;
will do what you want.

Unfortunately it only works if you qualify the column the_sum with a tablename or alias.
(But I heard you mention the Micro$oft word, and they tend to always use aliases anyway)
Maybe we could even extend the column search in the unqualified case ?

Andreas

At 18:00 23/11/00 +1300, John Huttley wrote:

1. Calculated fields in table definitions . eg.

Can't really do this - you might want to consider a view with an insert &
update rule. I'm not sure how flexible rules are and you may not be able to
write rules to make views functions like tables, but that is at least part
of their purpose I think.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

At 06:00 PM 11/23/00 +1300, John Huttley wrote:

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

...

These are _extraordinarily_ useful for application development.

If anyone has a way of bolting on any of these to 7.0, I'd be keen to hear

from

you.

Create a trigger on insert/update for this case...

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 12:28 PM 11/23/00 +0100, Zeugswetter Andreas SB wrote:

Reason: I want to know if any of these features are scheduled.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

This is currently easily done with a procedure that takes a tabletype

parameter

with the name the_sum returning the sum of a + b.

Create table test (
A Integer,
B integer
);

create function the_sum (test) returns integer as
'
begin;
return ($1.a + $1.b);
end;
' language 'plpgsql';

A select * won't return the_sum

create view test2 select A, B, A+B as the_sum from test;

will, though.

See, lots of ways to do it!

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

"john huttley" <john@mwk.co.nz> writes:

We've had parameterized triggers for years. Maybe you attach some
meaning to that term beyond what I do?

I'm referring to the manual that says functions used for triggers must
have no parameters and return a type Opaque.

The function has to be declared that way, but you can actually pass a
set of string parameters to it from the CREATE TRIGGER command. The
strings show up in some special variable or other inside the function.
(No, I don't know why it was done in that ugly way...) See the manual's
discussion of trigger programming.

regards, tom lane

Thanks for your help, everyone.

This is a summary of replies.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

This functionality can be achieved through the use of views.
Implementing the create table syntax may not be too hard,
but not in 7.1...

2 Parameterised Triggers

Functionality is there, just that the documentation gave the wrong implication.
An user manual example of using parameterised triggers to implement referential
integrity
would be welcome.

3. Stored Procedures returning a record set.

Dream on!

Regards

John

On Tue, 28 Nov 2000, John Huttley wrote:

3. Stored Procedures returning a record set.

Dream on!

This is something I would be really interested to see working. What are the
issues? my understanding is that it is technically feasible but too
complicated to add to PL/PGsql? it seems to me a basic service that needs
to be implemented soon, even if its just returning multiple rows of one
column...

- Andrew

Hi,

how long is PG7.1 already in beta testing? can it be released before Christmas day?
can PG7.1 will recover database from system crash?

Thanks,

XuYifeng

Hi,

how long is PG7.1 already in beta testing? can it be released before Christmas day?
can PG7.1 will recover database from system crash?

Thanks,

XuYifeng

At 04:17 PM 11/28/00 +0800, xuyifeng wrote:

Hi,

how long is PG7.1 already in beta testing? can it be released before Christmas day?
can PG7.1 will recover database from system crash?

This guy's a troll from the PHP Builder's site (at least, Tim Perdue and I suspect this
due to some posts he made in regard to Tim's SourceForge/Postgres article).

Since he's read Tim's article, and at least some of the follow-up posts (given that
he's posted responses himself), he should know by now that PG 7.1 is still in a pre-beta
state and won't be released before Christmas day. I also posted a fairly long answer
to a question Tim's posted at phpbuilder.com regarding recoverability and this guy's
undoubtably read it, too.

Have I forgotten anything, xuyifeng?

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

no doubt, I have touched some problems PG has, right? if PG is so good,
is there any necessary for the team to improve PG again?

Regards,
XuYifeng

----- Original Message -----
From: Don Baccus <dhogaza@pacifier.com>
To: xuyifeng <jamexu@telekbird.com.cn>; <pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 10:37 PM
Subject: Re: [HACKERS] beta testing version

At 11:15 PM 11/28/00 +0800, xuyifeng wrote:

no doubt, I have touched some problems PG has, right? if PG is so good,
is there any necessary for the team to improve PG again?

See? Troll...

The guy worships MySQL, just in case folks haven't made the connection.

I'm going to ignore him from now on, suggest others do the same, I'm sure
he'll go away eventually.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

you are complete wrong, if I don't like PG, I'll never go here or talk anything about PG, I don't care it.
I just want PG can be improved quickly, for me crash recover is very urgent problem,
otherewise PG is forced to stay on my desktop machine, We'll dare not move it to our Server,
I always see myself as a customer, customer is always right.

Regards,
XuYifeng

----- Original Message -----
From: Don Baccus <dhogaza@pacifier.com>
To: xuyifeng <jamexu@telekbird.com.cn>; <pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 11:16 PM
Subject: Re: [HACKERS] beta testing version

On Tue, Nov 28, 2000 at 02:04:01PM +1300, John Huttley wrote:

Thanks for your help, everyone.

This is a summary of replies.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

This functionality can be achieved through the use of views.

Using a view for this isn't quite the same functionality as a computed
field, from what I understand, since the calculation will be done at
SELECT time, rather than INSERT/UPDATE.

This can also be done with a trigger, which, while more cumbersome to
write, would be capable of doing the math at modification time.

Ross
--
Open source code is like a natural resource, it's the result of providing
food and sunshine to programmers, and then staying out of their way.
[...] [It] is not going away because it has utility for both the developers
and users independent of economic motivations. Jim Flynn, Sunnyvale, Calif.

no doubt, I have touched some problems PG has, right? if PG is so good,
is there any necessary for the team to improve PG again?

*rofl*

Good call Don :)

- Thomas

This is a summary of replies.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

This functionality can be achieved through the use of views.

Using a view for this isn't quite the same functionality as a computed
field, from what I understand, since the calculation will be done at
SELECT time, rather than INSERT/UPDATE.

I would expect the calculated field from above example to be calculated
during select time also, no ? You don't want to waste disk space with something
you can easily compute at runtime.

Andreas

I guess it depends on what you're using it for -- disk space
is cheap and
abundant anymore, I can see some advantages of having it
computed only once
rather than X times, where X is the number of SELECTs as that
could get
costly on really high traffic servers.. Costly not so much for simple
computations like that but more complex ones.

Once and for all forget the argument in database technology, that disk space
is cheap in regard to $/Mb. That is not the question. The issue is:
1. amout of rows you can cache
2. number of rows you can read from disk per second
(note that it is not pages/sec)
3. how many rows you can sort in memory

In the above sence disk space is one of the most expensive things in a
database system. Saving disk space where possible will gain you drastic
performance advantages.

Andreas

On Tue, 28 Nov 2000, xuyifeng wrote:

no doubt, I have touched some problems PG has, right? if PG is so good,
is there any necessary for the team to improve PG again?

There is always room for improvements for any software package ... whether
it be PgSQL, Linux, FreeBSD or PHPBuilder ... as ppl learn more,
understand more and come up with new techniques, things tend to get better
...

Regards,

XuYifeng

----- Original Message -----
From: Don Baccus <dhogaza@pacifier.com>
To: xuyifeng <jamexu@telekbird.com.cn>; <pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 10:37 PM
Subject: Re: [HACKERS] beta testing version

At 04:17 PM 11/28/00 +0800, xuyifeng wrote:

Hi,

how long is PG7.1 already in beta testing? can it be released before Christmas day?
can PG7.1 will recover database from system crash?

This guy's a troll from the PHP Builder's site (at least, Tim Perdue and I suspect this
due to some posts he made in regard to Tim's SourceForge/Postgres article).

Since he's read Tim's article, and at least some of the follow-up posts (given that
he's posted responses himself), he should know by now that PG 7.1 is still in a pre-beta
state and won't be released before Christmas day. I also posted a fairly long answer
to a question Tim's posted at phpbuilder.com regarding recoverability and this guy's
undoubtably read it, too.

Have I forgotten anything, xuyifeng?

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

On Tue, 28 Nov 2000, xuyifeng wrote:

you are complete wrong, if I don't like PG, I'll never go here or talk
anything about PG, I don't care it. I just want PG can be improved
quickly, for me crash recover is very urgent problem, otherewise PG is
forced to stay on my desktop machine, We'll dare not move it to our
Server, I always see myself as a customer, customer is always right.

except when they are wrong ...

... but, as for crash recover, the plan right now is that on Thursday, Dec
1st, 7.1 goes beta ... if you are so keen on the crash recovery stuff,
what I'd recommend is grab the snapshot, and work with that on your
machine, get used to the features that it presents and report any bugs you
find. Between beta and release, there will be bug fixes, but no features
added, so it makes for a relatively safe starting point. I wouldn't use
it in production (or, rather, I personally would, but it isn't something
I'd recommend for the faint of heart), but it will give you a base to
start from ...

release will be shortly into the new year, depending on what sorts of bugs
ppl report and how quickly they can be fixed ... if all goes well, Jan 1st
will be release date, but, from experience, we're looking at closer to jan
15th :)

Regards,

XuYifeng

----- Original Message -----
From: Don Baccus <dhogaza@pacifier.com>
To: xuyifeng <jamexu@telekbird.com.cn>; <pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 11:16 PM
Subject: Re: [HACKERS] beta testing version

At 11:15 PM 11/28/00 +0800, xuyifeng wrote:

no doubt, I have touched some problems PG has, right? if PG is so good,
is there any necessary for the team to improve PG again?

See? Troll...

The guy worships MySQL, just in case folks haven't made the connection.

I'm going to ignore him from now on, suggest others do the same, I'm sure
he'll go away eventually.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

On Tue, 28 Nov 2000, Hannu Krosing wrote:

xuyifeng wrote:

I just noticed this conversation so I have not followed all of it,
but you seem to have strange priorities

I just want PG can be improved quickly, for me crash recover is very urgent problem,

Crash avoidance is usually much more urgent, at least on production
servers.

Good call, but I kinda jumped to the conclusion that since PgSQL itself
isn't that crash prone, its his OS or his hardware that was the problem :0

On Tue, Nov 28, 2000 at 05:19:45PM +0100, Zeugswetter Andreas SB wrote:

I guess it depends on what you're using it for -- disk space
is cheap and
abundant anymore, I can see some advantages of having it
computed only once
rather than X times, where X is the number of SELECTs as that
could get
costly on really high traffic servers.. Costly not so much for simple
computations like that but more complex ones.

<snip good arguments about disk space>

As I said in my original post, my understanding of computed fields may
be in error. If they're computed at SELECT time, to avoid creating table
space, then a VIEW is exacly the right solution. However, it's easy to
come up with examples of complex calculations that it would be useful
to cache the results of, in the table. Then, computing at INSERT/UPDATE
is clearly the way to go.

So, having _both_ is the best thing.

Ross

I guess it depends on what you're using it for -- disk space is cheap and
abundant anymore, I can see some advantages of having it computed only once
rather than X times, where X is the number of SELECTs as that could get
costly on really high traffic servers.. Costly not so much for simple
computations like that but more complex ones.

Just playing the devil's advocate a bit.

-Mitch

----- Original Message -----
From: "Zeugswetter Andreas SB" <ZeugswetterA@wien.spardat.at>
To: "'Ross J. Reedstrom'" <reedstrm@rice.edu>;
<pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 7:50 AM
Subject: AW: [HACKERS] Please advise features in 7.1 (SUMMARY)

This is a summary of replies.

1. Calculated fields in table definitions . eg.

Create table test (
A Integer,
B integer,
the_sum As (A+B),
);

This functionality can be achieved through the use of views.

Using a view for this isn't quite the same functionality as a computed
field, from what I understand, since the calculation will be done at
SELECT time, rather than INSERT/UPDATE.

I would expect the calculated field from above example to be calculated
during select time also, no ? You don't want to waste disk space with

something

So, having _both_ is the best thing.

Absolutely, that's always what I meant -- we already have views and views
can do this type of stuff at SELECT time can't they? So it's not a change,
just an addition....

And the precalculated and stored on disk thing can be done with triggers.

Andreas

So, having _both_ is the best thing.

Absolutely, that's always what I meant -- we already have views and views
can do this type of stuff at SELECT time can't they? So it's not a change,
just an addition....

-Mitch

This is one of the not-so-stomped boxes running PostgreSQL -- I've never
restarted PostgreSQL on it since it was installed.

12:03pm up 122 days, 7:54, 1 user, load average: 0.08, 0.11, 0.09

I had some index corruption problems in 6.5.3 but since 7.0.X I haven't
heard so much as a peep from any PostgreSQL backend. It's superbly stable on
all my machines..

Damn good work guys.

-Mitch

----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: "Hannu Krosing" <hannu@tm.ee>
Cc: "xuyifeng" <jamexu@telekbird.com.cn>; <pgsql-hackers@postgresql.org>;
"Don Baccus" <dhogaza@pacifier.com>
Sent: Tuesday, November 28, 2000 8:53 AM
Subject: Re: [HACKERS] beta testing version

On Tue, 28 Nov 2000, Hannu Krosing wrote:

xuyifeng wrote:

I just noticed this conversation so I have not followed all of it,
but you seem to have strange priorities

I just want PG can be improved quickly, for me crash recover is very

urgent problem,

xuyifeng wrote:

I just noticed this conversation so I have not followed all of it,
but you seem to have strange priorities

I just want PG can be improved quickly, for me crash recover is very urgent problem,

Crash avoidance is usually much more urgent, at least on production
servers.

otherewise PG is forced to stay on my desktop machine, We'll dare not move it to our Server,

Why do you keep crashing your server ?

If your desktop crashes less often than your server you might exchange
them, no?

I always see myself as a customer, customer is always right.

I'd like to see myself as being always right too ;)

-------------------
Hannu

Mitch Vincent wrote:

This is one of the not-so-stomped boxes running PostgreSQL -- I've never
restarted PostgreSQL on it since it was installed.
12:03pm up 122 days, 7:54, 1 user, load average: 0.08, 0.11, 0.09
I had some index corruption problems in 6.5.3 but since 7.0.X I haven't
heard so much as a peep from any PostgreSQL backend. It's superbly stable on
all my machines..

I have a 6.5.x box at 328 days of active use.

Crash "recovery" seems silly to me. :-)

-Bop

--
Brought to you from boop!, the dual boot Linux/Win95 Compaq Presario 1625
laptop, currently running RedHat 6.1. Your bopping may vary.

At 03:25 PM 11/28/00 -0700, Ron Chmara wrote:

Mitch Vincent wrote:

This is one of the not-so-stomped boxes running PostgreSQL -- I've never
restarted PostgreSQL on it since it was installed.
12:03pm up 122 days, 7:54, 1 user, load average: 0.08, 0.11, 0.09
I had some index corruption problems in 6.5.3 but since 7.0.X I haven't
heard so much as a peep from any PostgreSQL backend. It's superbly stable on
all my machines..

I have a 6.5.x box at 328 days of active use.

Crash "recovery" seems silly to me. :-)

Well, not really ... but since our troll is a devoted MySQL user, it's a bit
of a red-herring anyway, at least as regards his own server.

You know, the one he's afraid to put Postgres on, but sleeps soundly at
night knowing the mighty bullet-proof MySQL with its full transaction
semantics, archive logging and recovery from REDO logs and all that
will save him? :)

Again ... he's a troll, not even a very entertaining one.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Tue, 28 Nov 2000, Don Baccus wrote:

At 03:25 PM 11/28/00 -0700, Ron Chmara wrote:

Mitch Vincent wrote:

This is one of the not-so-stomped boxes running PostgreSQL -- I've never
restarted PostgreSQL on it since it was installed.
12:03pm up 122 days, 7:54, 1 user, load average: 0.08, 0.11, 0.09
I had some index corruption problems in 6.5.3 but since 7.0.X I haven't
heard so much as a peep from any PostgreSQL backend. It's superbly stable on
all my machines..

I have a 6.5.x box at 328 days of active use.

Crash "recovery" seems silly to me. :-)

Well, not really ... but since our troll is a devoted MySQL user, it's a bit
of a red-herring anyway, at least as regards his own server.

You know, the one he's afraid to put Postgres on, but sleeps soundly at
night knowing the mighty bullet-proof MySQL with its full transaction
semantics, archive logging and recovery from REDO logs and all that
will save him? :)

Again ... he's a troll, not even a very entertaining one.

Or informed?

NO, I just tested how solid PgSQL is, I run a program busy inserting record into PG table, when I
suddenly pulled out power from my machine and restarted PG, I can not insert any record into database
table, all backends are dead without any respone (not core dump), note that I am using FreeBSD 4.2,
it's rock solid, it's not OS crash, it just losted power. We use WindowsNT and MSSQL on our production
server, before we accept MSSQL, we use this method to test if MSSQL can endure this kind of strik,
it's OK, all databases are safely recovered, we can continue our work. we are a stock exchange company,
our server are storing millilion $ finance number, we don't hope there are any problems in this case,
we are using UPS, but UPS is not everything, it you bet everything on UPS, you must be idiot.
I know you must be an avocation of PG, but we are professional customer, corporation user, we store critical
data into database, not your garbage data.

Regards,
XuYifeng

----- Original Message -----
From: Don Baccus <dhogaza@pacifier.com>
To: Ron Chmara <ron@opus1.com>; Mitch Vincent <mitch@venux.net>; <pgsql-hackers@postgresql.org>
Sent: Wednesday, November 29, 2000 6:58 AM
Subject: Re: [HACKERS] beta testing version

On Wed, Nov 29, 2000 at 09:59:34AM +0800, xuyifeng wrote:

NO, I just tested how solid PgSQL is, I run a program busy inserting
record into PG table, when I suddenly pulled out power from my machine ...

Nobody claims PostgreSQL is proof against power failures.

... We use WindowsNT and MSSQL on our production server,
before we accept MSSQL, we use this method to test if MSSQL can endure
this kind of strike, it's OK, all databases are safely recovered, we
can continue our work.

You got lucky. Period. MSSQL is not proof against power failures,
and neither is NTFS. In particular, that the database accepted
transactions afterward is far from proof that its files were not
corrupted.

Incompetent testers produce invalid tests. Invalid tests lead to
meaningless conclusions. Incompetent testers' employers suffer
from false confidence, and poor decision-making.

Nathan Myers
ncm@zembu.com

server, before we accept MSSQL, we use this method to test if MSSQL can

endure this kind of strik,

it's OK, all databases are safely recovered, we can continue our work. we

are a stock exchange company,

And how exactly did you test the integrity of your data? Unless every single
record has got at least a CRC stored somewhere, you won't be able AT ALL to
check for database integrity. The reports from NTFS and MSSQL internal
checking are meaningless for your data integrity.

We are doing this checksumming in our project, and already got a few nasty
surprises when the "CRC daemon" stumbled over a few corrupted records we
never would have discovered otherwise. Exactly this checksumming weeded out
our server alternatives; at present only PostgreSQL is left, was the most
reliable of all.

Horst

NO, I just tested how solid PgSQL is, I run a program busy inserting record into PG table, when I
suddenly pulled out power from my machine and restarted PG, I can not insert any record into database
table, all backends are dead without any respone (not core dump), note that I am using FreeBSD 4.2,
it's rock solid, it's not OS crash, it just losted power.

PostgreSQL Versions 7.0 and below have the potential of corruting indices when the system crashes.
The usual procedure would be to reindex the database. Hiroshi has written code to allow this.
It is weird, that your installation blocks. Have you checked the postmaster log ?

We use WindowsNT and MSSQL on our production
server, before we accept MSSQL, we use this method to test if MSSQL can endure this kind of strik,
it's OK, all databases are safely recovered, we can continue our work. we are a stock exchange company,
our server are storing millilion $ finance number, we don't hope there are any problems in this case,
we are using UPS, but UPS is not everything, it you bet everything on UPS, you must be idiot.
I know you must be an avocation of PG, but we are professional customer, corporation user, we store critical
data into database, not your garbage data.

Yes, this is a test I would also do before putting very sensitive data onto a particular brand of database.

Fortunately Version 7.1 of PostgreSQL will live up to your expectations in this area.

Andreas

xuyifeng wrote:

NO, I just tested how solid PgSQL is, I run a program busy inserting record into PG table, when I
suddenly pulled out power from my machine and restarted PG, I can not insert any record into database
table, all backends are dead without any respone (not core dump), note that I am using FreeBSD 4.2,
it's rock solid, it's not OS crash, it just losted power. We use WindowsNT and MSSQL on our production
server, before we accept MSSQL, we use this method to test if MSSQL can endure this kind of strik,
it's OK, all databases are safely recovered, we can continue our work.

The only way to safely recover them after a major crash would be
manual/supervised recovery from backups + logs

As not even NTFS is safe from power failures (I have lost an NTFS file
system a few times due to not
having an UPS) it is irrelevant if MSSQL is. Even if MSSQL is "crash
proof" (tm), how can you _prove_
your customers/superiors that the last N minutes of transactions were
not lost ?

If the DB is able to "continue your work" after the crash, you can of
course cover up the fact that the
crash even happened and blame the lost transactions on someone else when
they surface at the next audit ;)

Or just claim thet computer technology is so complicated that losing a
few transactions is normal - but
you could go on working ;) :~) ;-p

What you want for mission-critical data is replicated databases or at
least off-site logging, not "crash
recovery" at some arbitrarily chosen layer. You will need to recover
from the crash even if it destroys
the whole computer.

May I suggest another test for your NT/MSSQL setup - dont pull the plug
but change the input voltage
to 10 000 VAC, if this goes well, test vith 100 000 VAC ;)
This is also a scenario much less likely to be protected by an UPS than
power loss.

we are a stock exchange company,
our server are storing millilion $ finance number, we don't hope there are any problems in this case,
we are using UPS, but UPS is not everything, it you bet everything on UPS, you must be idiot.

So are you, if you bet everything on hoping that DB will do crash
recovery from any type of crash.

A common case of "crash" that may need to be recovered from is also a
human error , like typing drop database
at the wrong console;

I know you must be an avocation of PG, but we are professional customer, corporation user, we store critical
data into database, not your garbage data.

Then you'd better have a crash recovery infrastructure/procedures in
place and not hope that DB server
will do that automatically for you

--------------------
Hannu

From: "Ross J. Reedstrom" <reedstrm@rice.edu>

On Tue, Nov 28, 2000 at 05:19:45PM +0100, Zeugswetter Andreas SB wrote:

I guess it depends on what you're using it for -- disk space
is cheap and
abundant anymore, I can see some advantages of having it
computed only once
rather than X times, where X is the number of SELECTs as that
could get
costly on really high traffic servers.. Costly not so much for simple
computations like that but more complex ones.

<snip good arguments about disk space>

As I said in my original post, my understanding of computed fields may
be in error. If they're computed at SELECT time, to avoid creating table
space, then a VIEW is exacly the right solution. However, it's easy to
come up with examples of complex calculations that it would be useful
to cache the results of, in the table. Then, computing at INSERT/UPDATE
is clearly the way to go.

So, having _both_ is the best thing.

Ross

I'm new at this, but the view thing?
Isn't that just the same as:

create table test2 ( i1 int4, i2 int4);
...insert...
select i1,i2,i1+i2 from test2;

Magnus

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
PGP Key: http://www.genline.nu/mag_pgp.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

I don't have the same luck, sorry to say!

I am running Mandrake linux with OpenWall patched 2.2.17 kernel, dual p3
550Mhz, 1gb memory.
It's a really busy webserver that constantly is running with 10 in load.
Sometime it spikes to ~40-50 in load (the most we had was 114(!)).
I am running postgresql 7.0.2 (from the Mandrake rpm's).

One problem i have is that in one database we rapidly insert/delete in some
tables, and to maintain a good performance on that db, i have to run a
vacuum every hour(!).
I think that db has excessive indexes all over the place (if that could have
anything to do with it?).

Another other problem that is more severe is that the database "crashes"
(read: stops working), if i run psql and do a select it says
"001129.07:04:15.688 [25474] FATAL 1: Memory exhausted in AllocSetAlloc()"
and fails.
I have a cron script that watches postgres, and restarts it if it cant get a
select right.
It fails this way maybe once a day or two days.
I've searched the mailinglist archives for this problem, but it allways
seems that my problem doesn't fit the descriptions of the other ppl's
problem generating this error message.

I have not found the right time to upgrade to 7.0.3 yet, and i don't know if
that would solve anything.

Another problem i have is that i get "001128.12:58:01.248 [23444] FATAL 1:
Socket command type unknown" in my logs. I don't know if i get that from
the unix odbc driver, the remote windows odbc driver, or in unix standard db
connections.

I get "pq_recvbuf: unexpected EOF on client connection" alot too, but that i
think only indicates that the socket was closed in a not-so-nice way, and
that it is no "real" error.
It seems that the psql windows odbc driver is generating this.

The postmaster is running with these parameters: "-N 512 -B 1024 -i -o -S
4096"

But as a happy note i can tell you that we have a Linux box here (pentium
100, kernel 2.0.3x) that has near 1000 days uptime, and runs postgres 6.5.x.
It has never failed, not even a single time :)

Magnus Naeslund

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
PGP Key: http://www.genline.nu/mag_pgp.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

----- Original Message -----
From: "Mitch Vincent" <mitch@venux.net>
To: <pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 19:12
Subject: Re: [HACKERS] beta testing version

This is one of the not-so-stomped boxes running PostgreSQL -- I've never
restarted PostgreSQL on it since it was installed.

12:03pm up 122 days, 7:54, 1 user, load average: 0.08, 0.11, 0.09

I had some index corruption problems in 6.5.3 but since 7.0.X I haven't
heard so much as a peep from any PostgreSQL backend. It's superbly stable

on

all my machines..

Damn good work guys.

-Mitch

----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: "Hannu Krosing" <hannu@tm.ee>
Cc: "xuyifeng" <jamexu@telekbird.com.cn>; <pgsql-hackers@postgresql.org>;
"Don Baccus" <dhogaza@pacifier.com>
Sent: Tuesday, November 28, 2000 8:53 AM
Subject: Re: [HACKERS] beta testing version

On Tue, 28 Nov 2000, Hannu Krosing wrote:

xuyifeng wrote:

I just noticed this conversation so I have not followed all of it,
but you seem to have strange priorities

I just want PG can be improved quickly, for me crash recover is very

urgent problem,

Crash avoidance is usually much more urgent, at least on production
servers.

Good call, but I kinda jumped to the conclusion that since PgSQL itself
isn't that crash prone, its his OS or his hardware that was the problem

:0

----- Original Message -----
From: Zeugswetter Andreas SB <ZeugswetterA@wien.spardat.at>
Subject: AW: [HACKERS] beta testing version

NO, I just tested how solid PgSQL is, I run a program busy inserting record into PG table, when I
suddenly pulled out power from my machine and restarted PG, I can not insert any record into database
table, all backends are dead without any respone (not core dump), note that I am using FreeBSD 4.2,
it's rock solid, it's not OS crash, it just losted power.

PostgreSQL Versions 7.0 and below have the potential of corruting indices when the system crashes.
The usual procedure would be to reindex the database. Hiroshi has written code to allow this.
It is weird, that your installation blocks. Have you checked the postmaster log ?

REINDEX command failed, I have an unique index, PG claimed there were two records have same key,
but obviously, it's not the fact.

[snip]
Yes, this is a test I would also do before putting very sensitive data onto a particular brand of database.

Fortunately Version 7.1 of PostgreSQL will live up to your expectations in this area.

Andreas

Thanks,
---
XuYifeng

"Magnus Naeslund\(f\)" <mag@fbab.net> writes:

Another other problem that is more severe is that the database "crashes"
(read: stops working), if i run psql and do a select it says
"001129.07:04:15.688 [25474] FATAL 1: Memory exhausted in AllocSetAlloc()"
and fails.

That's odd. Does any select at all --- even, say, "SELECT 2+2" --- fail
like that, or just ones referencing a particular table, or maybe you
meant just one specific query?

Another problem i have is that i get "001128.12:58:01.248 [23444] FATAL 1:
Socket command type unknown" in my logs. I don't know if i get that from
the unix odbc driver, the remote windows odbc driver, or in unix standard db
connections.

Do any of your client applications complain that they're being
disconnected on? This might come from something not doing disconnection
cleanly, in which case the client probably wouldn't notice anything wrong.

I get "pq_recvbuf: unexpected EOF on client connection" alot too, but that i
think only indicates that the socket was closed in a not-so-nice way, and
that it is no "real" error.
It seems that the psql windows odbc driver is generating this.

That message is quite harmless AFAIK, although it'd be nice to clean up
the ODBC driver so that it disconnects in the approved fashion.

regards, tom lane

Is "if" clause support in PG?
for example:
"drop table aa if exist"
"insert into aa values(1) if not exists select * from aa where i=1"

I would like PG support it.
---
XuYifeng

----- Original Message -----
From: John Huttley <John@mwk.co.nz>
To: <pgsql-hackers@postgresql.org>
Sent: Tuesday, November 28, 2000 9:04 AM
Subject: [HACKERS] Please advise features in 7.1 (SUMMARY)

our server alternatives; at present only PostgreSQL is left, was the most
reliable of all.

mind i ask on which platform (Operating system) did you do your test,i'm
mostly used to linux but after i paid my computer (still 5 month
remaining),i want to get a used SGI box from reputable system and put NetBSD
as well as PostgreSQL on it (and maybe AolServer too,depending on the
threading model of NetBSD).

Alain Toussaint

On Thu, 30 Nov 2000, Thomas Lockhart wrote:

Is "if" clause support in PG?
for example:
"drop table aa if exist"
"insert into aa values(1) if not exists select * from aa where i=1"

No. afaict it is not in any SQL standard, so is unlikely to get much
attention from developers.

Plus, for that second one can't you just do:

INSERT INTO aa SELECT 1 WHERE NOT EXISTS (SELECT * FROM aa WHERE i=1);

- Andrew

Is "if" clause support in PG?
for example:
"drop table aa if exist"
"insert into aa values(1) if not exists select * from aa where i=1"

No. afaict it is not in any SQL standard, so is unlikely to get much
attention from developers.

- Thomas

At 05:24 AM 11/30/00 +0000, Thomas Lockhart wrote:

Is "if" clause support in PG?
for example:
"drop table aa if exist"
"insert into aa values(1) if not exists select * from aa where i=1"

No. afaict it is not in any SQL standard, so is unlikely to get much
attention from developers.

The insert, at least, can be written in standard SQL anyway...

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

v7.1 should improve crash recovery for situations like this ... you'll
still have to do a recovery of the data on corruption of this magnitude,
but at least with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of the wall
...

On Wed, 29 Nov 2000, xuyifeng wrote:

NO, I just tested how solid PgSQL is, I run a program busy inserting record into PG table, when I
suddenly pulled out power from my machine and restarted PG, I can not insert any record into database
table, all backends are dead without any respone (not core dump), note that I am using FreeBSD 4.2,
it's rock solid, it's not OS crash, it just losted power. We use WindowsNT and MSSQL on our production
server, before we accept MSSQL, we use this method to test if MSSQL can endure this kind of strik,
it's OK, all databases are safely recovered, we can continue our work. we are a stock exchange company,
our server are storing millilion $ finance number, we don't hope there are any problems in this case,
we are using UPS, but UPS is not everything, it you bet everything on UPS, you must be idiot.
I know you must be an avocation of PG, but we are professional customer, corporation user, we store critical
data into database, not your garbage data.

Regards,
XuYifeng

----- Original Message -----
From: Don Baccus <dhogaza@pacifier.com>
To: Ron Chmara <ron@opus1.com>; Mitch Vincent <mitch@venux.net>; <pgsql-hackers@postgresql.org>
Sent: Wednesday, November 29, 2000 6:58 AM
Subject: Re: [HACKERS] beta testing version

At 03:25 PM 11/28/00 -0700, Ron Chmara wrote:

Mitch Vincent wrote:

This is one of the not-so-stomped boxes running PostgreSQL -- I've never
restarted PostgreSQL on it since it was installed.
12:03pm up 122 days, 7:54, 1 user, load average: 0.08, 0.11, 0.09
I had some index corruption problems in 6.5.3 but since 7.0.X I haven't
heard so much as a peep from any PostgreSQL backend. It's superbly stable on
all my machines..

I have a 6.5.x box at 328 days of active use.

Crash "recovery" seems silly to me. :-)

Well, not really ... but since our troll is a devoted MySQL user, it's a bit
of a red-herring anyway, at least as regards his own server.

You know, the one he's afraid to put Postgres on, but sleeps soundly at
night knowing the mighty bullet-proof MySQL with its full transaction
semantics, archive logging and recovery from REDO logs and all that
will save him? :)

Again ... he's a troll, not even a very entertaining one.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

At 07:02 PM 11/30/00 -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery for situations like this ... you'll
still have to do a recovery of the data on corruption of this magnitude,
but at least with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of the wall

No, WAL won't help if an actual database file is corrupted, say by a
disk drive hosing a block or portion thereof with zeros. WAL-based
recovery at startup works on an intact database.

Still, in the general case you need real backup and recovery tools.
Then you can apply archives of REDOs to a backup made of a snapshot
and rebuild up to the last transaction. As opposed to your last
pg_dump.

So what about mirroring (RAID 1)? As the docs tell ya, that protects
you against one drive failing but not against power failure, which can
cause bad data to be written to both mirrors if both are actively
writing when the plug is pulled.

Power failures are evil, face it! :)

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Thu, Nov 30, 2000 at 07:02:01PM -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery ...
... with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of
the wall.

Please do not propagate falsehoods like the above. It creates
unsatisfiable expectations, and leads people to fail to take
proper precautions and recovery procedures.

After a power outage on an active database, you may have corruption
at low levels of the system, and unless you have enormous redundancy
(and actually use it to verify everything) the corruption may go
undetected and result in (subtly) wrong answers at any future time.

The logging in 7.1 protects transactions against many sources of
database crash, but not necessarily against OS crash, and certainly
not against power failure. (You might get lucky, or you might just
think you were lucky.) This is the same as for most databases; an
embedded database that talks directly to the hardware might be able
to do better.

Nathan Myers
ncm@zembu.com

On Thu, 30 Nov 2000, Don Baccus wrote:

At 07:02 PM 11/30/00 -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery for situations like this ... you'll
still have to do a recovery of the data on corruption of this magnitude,
but at least with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of the wall

No, WAL won't help if an actual database file is corrupted, say by a
disk drive hosing a block or portion thereof with zeros. WAL-based
recovery at startup works on an intact database.

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

On Thu, Nov 30, 2000 at 07:47:08PM -0400, The Hermit Hacker wrote:

On Thu, 30 Nov 2000, Don Baccus wrote:

At 07:02 PM 11/30/00 -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery for situations like this ... you'll
still have to do a recovery of the data on corruption of this magnitude,
but at least with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of the wall

No, WAL won't help if an actual database file is corrupted, say by a
disk drive hosing a block or portion thereof with zeros. WAL-based
recovery at startup works on an intact database.

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

Nathan Myers
ncm

On Thu, 30 Nov 2000, Nathan Myers wrote:

On Thu, Nov 30, 2000 at 07:47:08PM -0400, The Hermit Hacker wrote:

On Thu, 30 Nov 2000, Don Baccus wrote:

At 07:02 PM 11/30/00 -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery for situations like this ... you'll
still have to do a recovery of the data on corruption of this magnitude,
but at least with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of the wall

No, WAL won't help if an actual database file is corrupted, say by a
disk drive hosing a block or portion thereof with zeros. WAL-based
recovery at startup works on an intact database.

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

would love to hear them ... I'm always opening to having my
misunderstandings corrected ...

On Thu, 30 Nov 2000, Nathan Myers wrote:

On Thu, Nov 30, 2000 at 07:47:08PM -0400, The Hermit Hacker wrote:

On Thu, 30 Nov 2000, Don Baccus wrote:

At 07:02 PM 11/30/00 -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery for situations like this ... you'll
still have to do a recovery of the data on corruption of this magnitude,
but at least with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of the wall

No, WAL won't help if an actual database file is corrupted, say by a
disk drive hosing a block or portion thereof with zeros. WAL-based
recovery at startup works on an intact database.

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

*YAWN*

On Thu, Nov 30, 2000 at 05:37:58PM -0800, Mitch Vincent wrote:

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

Why? If we're not talking hardware damage and you have a dump made
sometime previous to the crash, why wouldn't that work to restore the
database? I've had to restore a corrupted database from a dump before,
there wasn't any hardware damage, the database (more specifically the
indexes) were corrupted. Of course WAL wasn't around but I don't see
why this wouldn't work...

I posted a more detailed explanation a few minutes ago, but
it appears to have been eaten by the mailing list server.

I won't re-post the explanations that you all have seen over the
last two days, about disk behavior during a power outage; they're
in the archives (I assume -- when last I checked, web access to it
didn't work). Suffice to say that if you pull the plug, there is
just too much about the state of the disks that is unknown.

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

To get replaying we need an "update log", something that might be
in 7.2 if somebody does a lot of work.

Note I'm not saying you're wrong, just asking that you explain your
comment a little more. If WAL can't be used to help recover from
crashes where database corruption occurs, what good is it?

The WAL is a performance optimization for the current recovery
capabilities, which assume uncorrupted table files. It protects
against those database server crashes that happen not to corrupt
the table files (i.e. most). It doesn't protect against corruption
of the tables, by bugs in PG or in the OS or from "hardware events".
It also doesn't protect against OS crashes that result in
write-buffered sectors not having been written before the crash.
Practically, this means that WAL file entries older than a few
seconds are not useful for much.

In general, it's foolish to expect a single system to store very
valuable data with much confidence. To get full recoverability,
you need a "hot failover" system duplicating your transactions in
real time. (Even then, you're vulnerable to application-level
mistakes.)

Nathan Myers
ncm@zembu.com

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

Why? If we're not talking hardware damage and you have a dump made sometime
previous to the crash, why wouldn't that work to restore the database? I've
had to restore a corrupted database from a dump before, there wasn't any
hardware damage, the database (more specifically the indexes) were
corrupted. Of course WAL wasn't around but I don't see why this wouldn't
work...

Note I'm not saying you're wrong, just asking that you explain your comment
a little more. If WAL can't be used to help recover from crashes where
database corruption occurs, what good is it?

-Mitch

At 05:15 PM 11/30/00 -0800, Nathan Myers wrote:

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

Actually, the dump doesn't record tuple OIDs (unless you specifically
ask for them), it just dumps source sql. When this gets reloaded
you get an equivalent database, but not the same database, that you
started out with.

That's why I've presumed you can't run the WAL against it.

If you and I are wrong I'd love to be surprised!

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Thu, 30 Nov 2000, Nathan Myers wrote:

On Thu, Nov 30, 2000 at 07:02:01PM -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery ...
... with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of
the wall.

Please do not propagate falsehoods like the above. It creates
unsatisfiable expectations, and leads people to fail to take
proper precautions and recovery procedures.

After a power outage on an active database, you may have corruption
at low levels of the system, and unless you have enormous redundancy
(and actually use it to verify everything) the corruption may go
undetected and result in (subtly) wrong answers at any future time.

The logging in 7.1 protects transactions against many sources of
database crash, but not necessarily against OS crash, and certainly
not against power failure. (You might get lucky, or you might just
think you were lucky.) This is the same as for most databases; an
embedded database that talks directly to the hardware might be able
to do better.

We're talking about transaction logging here ... nothing gets written to
it until completed ... if I take a "known to be clean" backup from the
night before, restore that and then run through the transaction logs, my
data should be clean, unless my tape itself is corrupt. If the power goes
off half way through a write to the log, then that transaction wouldn't be
marked as completed and won't roll into the restore ...

if a disk goes corrupt, I'd expect that the redo log would possibly have a
problem with corruption .. but if I pull the plug, unless I've somehow
damaged the disk, I would expect my redo log to be clean *and*, unless
Vadim totally messed something up, if there is any corruption in the redo
log, I'd expect that restoring from it would generate from red flags ...

At 03:35 PM 11/30/00 -0800, Nathan Myers wrote:

On Thu, Nov 30, 2000 at 07:02:01PM -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery ...
... with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of
the wall.

Please do not propagate falsehoods like the above. It creates
unsatisfiable expectations, and leads people to fail to take
proper precautions and recovery procedures.

Yeah, I posted similar stuff to the PHPbuilder forum in regard to
PG.

The logging in 7.1 protects transactions against many sources of
database crash, but not necessarily against OS crash, and certainly
not against power failure. (You might get lucky, or you might just
think you were lucky.) This is the same as for most databases; an
embedded database that talks directly to the hardware might be able
to do better.

Let's put it this way ... Oracle, a transaction-safe DB with REDO
logging, has for a very long time implemented disk mirroring. Now,
why would they do that if you could pull the plug on the processor
and depend on REDO logging to save you?

And even then you're expected to provide adequate power backup to
enable clean shutdown.

The real safety you get is that your battery sez "we need to shut
down!" but has enough power to let you. Transactions in progress
aren't logged, but everything else can tank cleanly, and your DB is
in a consistent state.

Mirroring protects you against (some) disk drive failures (but not
those that are transparent to the RAID controller/driver - if your
drive writes crap to the primary side of the mirror and no errors
are returned to the hardware/driver, the other side of the mirror
can faithfully reproduce them on the mirror!)

But since drives contain bearings and such that are much more likely
to fail than electronics (good electronics and good designs, at least),
mechanical failure's more likely and will be known to whatever is driving
the drive. And you're OK then...

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Thu, 30 Nov 2000, Nathan Myers wrote:

After a power outage on an active database, you may have corruption
at low levels of the system, and unless you have enormous redundancy
(and actually use it to verify everything) the corruption may go
undetected and result in (subtly) wrong answers at any future time.

Nathan, why are you so hostile against postgres? Is there an ax to grind?

The conditions under which WAL will completely recover your database:
1) OS guarantees complete ordering of fsync()'d writes. (i.e. having two
blocks A and B, A is fsync'd before B, it could NOT happen that B is on
disk but A is not).
2) on boot recovery, OS must not corrupt anything that was fsync'd.

Rule 1) is met by all unixish OSes in existance. Rule 2 is met by some
filesystems, such as reiserfs, tux2, and softupdates.

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

True for current way of backing up - ie saving data in "external"
(sql) format. But there is another way - saving data files in their
natural (binary) format. WAL records may be applyed to
such dump, right?

To get replaying we need an "update log", something that might be

What did you mean by "update log"?
Are you sure that WAL is not "update log" ? -:)

in 7.2 if somebody does a lot of work.

Note I'm not saying you're wrong, just asking that you explain your
comment a little more. If WAL can't be used to help recover from
crashes where database corruption occurs, what good is it?

The WAL is a performance optimization for the current recovery
capabilities, which assume uncorrupted table files. It protects
against those database server crashes that happen not to corrupt
the table files (i.e. most). It doesn't protect against corruption
of the tables, by bugs in PG or in the OS or from "hardware events".
It also doesn't protect against OS crashes that result in
write-buffered sectors not having been written before the crash.
Practically, this means that WAL file entries older than a few
seconds are not useful for much.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Even now, without BAR, WAL entries become unuseful only after checkpoints
(and I wouldn't recomend to create them each few seconds -:)). WAL based
BAR will require archiving of log records.

Vadim

On Fri, Dec 01, 2000 at 12:00:12AM -0400, The Hermit Hacker wrote:

On Thu, 30 Nov 2000, Nathan Myers wrote:

On Thu, Nov 30, 2000 at 07:02:01PM -0400, The Hermit Hacker wrote:

v7.1 should improve crash recovery ...
... with the WAL stuff that Vadim is producing, you'll be able to
recover up until the point that the power cable was pulled out of
the wall.

Please do not propagate falsehoods like the above. It creates
unsatisfiable expectations, and leads people to fail to take
proper precautions and recovery procedures.

After a power outage on an active database, you may have corruption
at low levels of the system, and unless you have enormous redundancy
(and actually use it to verify everything) the corruption may go
undetected and result in (subtly) wrong answers at any future time.

The logging in 7.1 protects transactions against many sources of
database crash, but not necessarily against OS crash, and certainly
not against power failure. (You might get lucky, or you might just
think you were lucky.) This is the same as for most databases; an
embedded database that talks directly to the hardware might be able
to do better.

We're talking about transaction logging here ... nothing gets written
to it until completed ... if I take a "known to be clean" backup from
the night before, restore that and then run through the transaction
logs, my data should be clean, unless my tape itself is corrupt. If
the power goes off half way through a write to the log, then that
transaction wouldn't be marked as completed and won't roll into the
restore ...

Sorry, wrong. First, the only way that your backups could have any
relationship with the transaction logs is if they are copies of the
raw table files with the database shut down, rather than the normal
"snapshot" backup.

Second, the transaction log is not, as has been noted far too frequently
for Vince's comfort, really written atomically. The OS has promised
to write it atomically, and given the opportunity, it will. If you pull
the plug, all promises are broken.

if a disk goes corrupt, I'd expect that the redo log would possibly
have a problem with corruption .. but if I pull the plug, unless I've
somehow damaged the disk, I would expect my redo log to be clean
*and*, unless Vadim totally messed something up, if there is any
corruption in the redo log, I'd expect that restoring from it would
generate from red flags ...

You have great expectations, but nobody has done the work to satisfy
them, so when you pull the plug, I'd expect that you will be left
in the dark, alone and helpless.

Vadim has done an excellent job on what he set out to do: optimize
transaction processing. Designing and implementing a factor-of-twenty
speed improvement on a professional-quality database engine demanded
great effort and expertise. To complain that he hasn't also done
a lot of other stuff would be petty.

Nathan Myers
ncm@zembu.com

On Fri, Dec 01, 2000 at 01:54:23AM -0500, Alex Pilosov wrote:

On Thu, 30 Nov 2000, Nathan Myers wrote:

After a power outage on an active database, you may have corruption
at low levels of the system, and unless you have enormous redundancy
(and actually use it to verify everything) the corruption may go
undetected and result in (subtly) wrong answers at any future time.

Nathan, why are you so hostile against postgres? Is there an ax to grind?

Alex, please don't invent enemies. It's clear what important features
PostgreSQL still lacks; over the next several releases these features
will be implemented, at great expense. PostgreSQL is useful and usable
now, given reasonable precautions and expectations. In the future it
will satisfy greater (albeit still reasonable) expectations.

The conditions under which WAL will completely recover your database:

1) OS guarantees complete ordering of fsync()'d writes. (i.e. having two
blocks A and B, A is fsync'd before B, it could NOT happen that B is on
disk but A is not).
2) on boot recovery, OS must not corrupt anything that was fsync'd.

Rule 1) is met by all unixish OSes in existance. Rule 2 is met by some
filesystems, such as reiserfs, tux2, and softupdates.

No. The OS asks the disk to write blocks in a certain order, but
disks normally reorder writes. Not only that; as noted earlier,
typical disks report the write completed long before the blocks
actually hit the disk.

A logging file system protects against the simpler forms of OS crash,
where the OS data-structure corruption is noticed before any more disk
writes are scheduled. It can't (by itself) protect against disk
errors. For critical applications, you must supply that protection
yourself, with (e.g.) battery-backed mirroring.

The logging in 7.1 protects transactions against many sources of
database crash, but not necessarily against OS crash, and certainly
not against power failure. (You might get lucky, or you might just
think you were lucky.) This is the same as for most databases; an
embedded database that talks directly to the hardware might be able
to do better.

The best possible database code can't overcome a broken OS or a broken
disk. It would be unreasonable to expect otherwise.

Nathan Myers
ncm@zembu.com

Date: Fri, 1 Dec 2000 01:54:23 -0500 (EST)
From: Alex Pilosov <alex@pilosoft.com>

On Thu, 30 Nov 2000, Nathan Myers wrote:

After a power outage on an active database, you may have corruption
at low levels of the system, and unless you have enormous redundancy
(and actually use it to verify everything) the corruption may go
undetected and result in (subtly) wrong answers at any future time.

Nathan, why are you so hostile against postgres? Is there an ax to grind?

I don't think he is being hostile (I work with him, so I know that he
is generally pro-postgres).

The conditions under which WAL will completely recover your database:
1) OS guarantees complete ordering of fsync()'d writes. (i.e. having two
blocks A and B, A is fsync'd before B, it could NOT happen that B is on
disk but A is not).
2) on boot recovery, OS must not corrupt anything that was fsync'd.

Rule 1) is met by all unixish OSes in existance. Rule 2 is met by some
filesystems, such as reiserfs, tux2, and softupdates.

I think you are missing his main point, which he stated before, which
is that modern disk hardware is both smarter and stupider than most
people realize.

Some disks cleverly accept writes into a RAM cache, and return a
completion signal as soon as they have done that. They then feel free
to reorder the writes to magnetic media as they see fit. This
significantly helps performance. However, it means that all bets off
on a sudden power loss.

Your rule 1 is met at the OS level, but it is not met at the physical
drive level. The fact that the OS guarantees ordering of fsync()'d
writes means little since the drive is capable of reordering writes
behind the back of the OS.

At least with IDE, it is possible to tell the drive to disable this
sort of caching and reordering. However, GNU/Linux, at least, does
not do this. After all, doing it would hurt performance, and would
move us back to the old days when operating systems had to care a
great deal about disk geometry.

I expect that careful attention to the physical disks you purchase can
help you avoid these problems. For example, I would hope that EMC
disk systems handle power loss gracefully. But if you buy ordinary
off the shelf PC hardware, you really do need to arrange for a UPS,
and some sort of automatic shutdown if the UPS is running low.
Otherwise, although the odds are certainly with you, there is no 100%
guarantee that a busy database will survive a sudden power outage.

Ian

On Thu, Nov 30, 2000 at 11:06:31PM -0800, Vadim Mikheev wrote:

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

True for current way of backing up - ie saving data in "external"
(sql) format. But there is another way - saving data files in their
natural (binary) format. WAL records may be applyed to
such dump, right?

But (AIUI) you can only safely/usefully copy those files when the
database is shut down.

Many people hope to run PostgreSQL 24x7x365. With vacuuming, you
might just as well shut down afterward; but when that goes away
(in 7.2?), when will you get the chance to take your backups?
Clearly we need either another form of snapshot backup that can
be taken with the database running, and compatible with the
current WAL (or some variation on it); or, we need another kind
of log, in addition to the WAL.

To get replaying we need an "update log", something that might be
in 7.2 if somebody does a lot of work.

What did you mean by "update log"?
Are you sure that WAL is not "update log" ? -:)

No, I'm not sure. I think it's possible that a new backup utility
could be written to make a hot backup which could be restored and
then replayed using the current WAL format. It might be easier to
add another log which could be replayed against the existing form
of backups. That last is what I called the "update log".

The point is, WAL now does one job superbly: maintain a consistent
on-disk database image. Asking it to do something else, such as
supporting hot BAR, could interfere with it doing its main job.
Of course, only the person who implements hot BAR can say.

Nathan Myers
ncm@zembu.com

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

would love to hear them ... I'm always opening to having my
misunderstandings corrected ...

Only what has been transferred off site can be considered safe.
But: all the WAL improvements serve to reduce the probability that
you 1. need to restore and 2. need to restore from offsite backups.

If you need to restore from offsite backup you loose transactions
unless you transfer the WAL synchronously with every commit.

Andreas

At 00:55 1/12/00 -0800, Nathan Myers wrote:

On Thu, Nov 30, 2000 at 11:06:31PM -0800, Vadim Mikheev wrote:

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

True for current way of backing up - ie saving data in "external"
(sql) format. But there is another way - saving data files in their
natural (binary) format. WAL records may be applyed to
such dump, right?

But (AIUI) you can only safely/usefully copy those files when the
database is shut down.

This is not true; the way Vadim has implemeted WAL is to write a series of
files of fixed size. When all transactions that have records in one file
have completed, that file is (currently) deleted. When BAR is going, the
files will be archived.

The only circumstance in which this strategy will fail is if there are a
large number of intensive long-standing single transactions - which is
unlikely (not to mention bad practice).

As a result of this, BAR will just need to take a snapshot of the database
and apply the logs (basically like a very extended recovery process).

You have raised some interesting issues regrading write-order etc. Can we
assume that when fsync *returns*, all records are written - though not
necessarily in the order that the IO's were executed?

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

At 11:06 PM 11/30/00 -0800, Vadim Mikheev wrote:

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

True for current way of backing up - ie saving data in "external"
(sql) format. But there is another way - saving data files in their
natural (binary) format. WAL records may be applyed to
such dump, right?

Right. That's what's missing in PG 7.1, the existence of tools to
make such backups.

Probably the best answer to the "what does WAL get us, if it doesn't
get us full recoverability" questions is to simply say "it's a prerequisite
to getting full recoverability, PG 7.1 sets the foundation and later
work will get us there".

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 12:30 AM 12/1/00 -0800, Ian Lance Taylor wrote:

For example, I would hope that EMC
disk systems handle power loss gracefully.

They must, their marketing literature says so :)

But if you buy ordinary
off the shelf PC hardware, you really do need to arrange for a UPS,
and some sort of automatic shutdown if the UPS is running low.

Which is what disk subsystems like those from EMC do for you. They've
got build-in battery backup that lets them guarantee (assuming the
hardware's working right) that in the case of a power outage, all blocks
the operating system thinks have been written will in actuality be written
before the disk subsystem powers itself down.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 12:55 AM 12/1/00 -0800, Nathan Myers wrote:

Many people hope to run PostgreSQL 24x7x365. With vacuuming, you
might just as well shut down afterward; but when that goes away
(in 7.2?), when will you get the chance to take your backups?
Clearly we need either another form of snapshot backup that can
be taken with the database running, and compatible with the
current WAL (or some variation on it); or, we need another kind
of log, in addition to the WAL.

Vadim's not ignorant of such matters, when he says "make a copy
of the files" he's not talking about using tar on a running
database. BAR tools are needed, as Vadim has pointed out here in
the past.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

As for replaying logs against a restored snapshot dump... AIUI, a
dump records tuples by OID, but the WAL refers to TIDs. Therefore,
the WAL won't work as a re-do log to recover your transactions
because the TIDs of the restored tables are all different.

True for current way of backing up - ie saving data in "external"
(sql) format. But there is another way - saving data files in their
natural (binary) format. WAL records may be applyed to
such dump, right?

But (AIUI) you can only safely/usefully copy those files when the
database is shut down.

No. You can read/save datafiles at any time. But block reads must be
"atomic" - no one should be able to change any part of a block while
we read it. Cp & tar are probably not suitable for this, but internal
BACKUP command could do this.

Restoring from such backup will like recovering after pg_ctl -m i stop: all
data blocks are consistent and WAL records may be applyed to them.

Many people hope to run PostgreSQL 24x7x365. With vacuuming, you
might just as well shut down afterward; but when that goes away
(in 7.2?), when will you get the chance to take your backups?

Ability to shutdown 7.2 will be preserved -:))
But it's not required for backup.

To get replaying we need an "update log", something that might be
in 7.2 if somebody does a lot of work.

What did you mean by "update log"?
Are you sure that WAL is not "update log" ? -:)

No, I'm not sure. I think it's possible that a new backup utility
could be written to make a hot backup which could be restored and
then replayed using the current WAL format. It might be easier to
add another log which could be replayed against the existing form
of backups. That last is what I called the "update log".

Consistent read of data blocks is easier to implement, sure.

The point is, WAL now does one job superbly: maintain a consistent
on-disk database image. Asking it to do something else, such as
supporting hot BAR, could interfere with it doing its main job.
Of course, only the person who implements hot BAR can say.

There will be no interference because of BAR will not ask WAL to do
anything else it does right now - redo-ing changes.

Vadim

On Fri, Dec 01, 2000 at 06:39:57AM -0800, Don Baccus wrote:

Probably the best answer to the "what does WAL get us, if it doesn't
get us full recoverability" questions is to simply say "it's a
prerequisite to getting full recoverability, PG 7.1 sets the foundation
and later work will get us there".

Not to quibble, but for most of us, the answer to Don's question is:
"It gives a ~20x speedup over 7.0." That's pretty valuable to some of us.
If it turns out to be useful for other stuff, that's gravy.

Nathan Myers
ncm@zembu.com

On Fri, Dec 01, 2000 at 10:01:15AM +0100, Zeugswetter Andreas SB wrote:

No, WAL does help, cause you can then pull in your last dump and recover
up to the moment that power cable was pulled out of the wall ...

False, on so many counts I can't list them all.

would love to hear them ... I'm always opening to having my
misunderstandings corrected ...

Only what has been transferred off site can be considered safe.
But: all the WAL improvements serve to reduce the probability that
you 1. need to restore and 2. need to restore from offsite backups.

If you need to restore from offsite backup you loose transactions
unless you transfer the WAL synchronously with every commit.

Currently the only way to avoid losing those transactions is by
replicating transactions at the application layer. That is, the
application talks to two different database instances, and enters
transactions into both. That's pretty hard to retrofit into an
existing application, so you'd really rather have replication in
the database. Of course, that's something PostgreSQL, Inc. is also
working on.

Nathan Myers
ncm@zembu.com

At 11:02 AM 12/1/00 -0800, Nathan Myers wrote:

On Fri, Dec 01, 2000 at 06:39:57AM -0800, Don Baccus wrote:

Probably the best answer to the "what does WAL get us, if it doesn't
get us full recoverability" questions is to simply say "it's a
prerequisite to getting full recoverability, PG 7.1 sets the foundation
and later work will get us there".

Not to quibble, but for most of us, the answer to Don's question is:
"It gives a ~20x speedup over 7.0." That's pretty valuable to some of us.
If it turns out to be useful for other stuff, that's gravy.

Oh, but given that power failures eat disks anyway, you can just run PG 7.0
with -F and be just as fast as PG 7.1, eh? With no theoretical loss in
safety? Where's your faith in all that doom and gloom you've been
spreading? :) :)

You're right, of course, we'll get roughly -F performance while maintaining
a much more comfortable level of risk than you get with -F.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Fri, Dec 01, 2000 at 09:13:28PM +1100, Philip Warner wrote:

You have raised some interesting issues regrading write-order etc. Can we
assume that when fsync *returns*, all records are written - though not
necessarily in the order that the IO's were executed?

Not with ordinary disks. With a battery-backed disk server, yes.

Nathan Myers
ncm@zembu.com

ncm@zembu.com (Nathan Myers) writes:

On Fri, Dec 01, 2000 at 09:13:28PM +1100, Philip Warner wrote:

You have raised some interesting issues regrading write-order etc. Can we
assume that when fsync *returns*, all records are written - though not
necessarily in the order that the IO's were executed?

Not with ordinary disks. With a battery-backed disk server, yes.

I think the real point of this discussion is that there's no such thing
as an ironclad guarantee. That's why people make backups.

All we can do is the best we can ;-). In that light, I think it's
reasonable for Postgres to proceed on the assumption that fsync does
what it claims to do, ie, all blocks are written when it returns.
We can't realistically expect to persuade a disk controller that
reorders writes to stop doing so. We can, however, expect that we've
minimized the probability of failures induced by anything other than
disk hardware failure or power failure.

regards, tom lane

At 11:09 AM 12/1/00 -0800, Nathan Myers wrote:

On Fri, Dec 01, 2000 at 10:01:15AM +0100, Zeugswetter Andreas SB wrote:

If you need to restore from offsite backup you loose transactions
unless you transfer the WAL synchronously with every commit.

Currently the only way to avoid losing those transactions is by
replicating transactions at the application layer. That is, the
application talks to two different database instances, and enters
transactions into both. That's pretty hard to retrofit into an
existing application, so you'd really rather have replication in
the database. Of course, that's something PostgreSQL, Inc. is also
working on.

Recovery alone isn't quite that difficult. You don't need to instantiate
your database instance until you need to apply the archived transactions,
i.e. after catastrophic failure destroys your db server.

You need to do two things:

1. Transmit a consistent (known-state) snapshot of the database offsite.
2. Synchronously tranfer the WAL as part of every commit (question, do
wait to log a "commit" locally until after the remote site acks that
it got the WAL?)

Then you take a new machine, build a database out of the snapshot, and
apply the archived redo logs and off you go. If you get tired of saving
oodles of redo archives, you make a new snapshot and accumulate the
WAL from that point forward.

Of course, that's not a fast failover solution. The scenario you describe
leads to being able to quickly switch over to a backup server when the
primary server fails. Much better for 24/7/365-style computing.

Exactly what is PostgreSQL, Inc doing in this area? I've not seen
discussions about it here, and the two of the three most active developers
(Jan and Tom) work for Great Bridge, not PostgreSQL, Inc...

I should think Vadim should play a large role in any effort to add WAL-based
replication to Postgres.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Fri, Dec 01, 2000 at 08:10:40AM -0800, Vadim Mikheev wrote:

... a new backup utility
could be written to make a hot backup which could be restored and
then replayed using the current WAL format. It might be easier to
add another log which could be replayed against the existing form
of backups. That last is what I called the "update log".

Consistent read of data blocks is easier to implement, sure.

The point is, WAL now does one job superbly: maintain a consistent
on-disk database image. Asking it to do something else, such as
supporting hot BAR, could interfere with it doing its main job.
Of course, only the person who implements hot BAR can say.

There will be no interference because of BAR will not ask WAL to do
anything else it does right now - redo-ing changes.

The interference I meant is that the current WAL file format is designed
for its current job. For BAR, you would be better-served by a more
compact format, so you need not archive your logs so frequently.
(The size of the WAL doesn't matter much because you can rotate them
very quickly.) A more compact format is also better as a basis for
replication, to minimize network traffic. To compress the WAL would
hurt performance -- but adding performance was the point of the WAL.

A log encoded at a much higher semantic level could be much more
compact, but wouldn't be useful as a WAL because it describes
differences from a snapshot backup, not from the current table
file contents.

Thus, I'm not saying that you can't implement both WAL and hot BAR
using the same log; rather, it's just not _obviously_ the best way to
do it.

Nathan Myers
ncm@zembu.com

Ok, this has peaked my interest in learning exactly what WAL is and what it
does... I don't see any in-depth explanation of WAL on the postgresql.org
site, can someone point me to some documentation? (if any exists, that is).

Thanks!

-Mitch

----- Original Message -----
From: "Nathan Myers" <ncm@zembu.com>
To: <pgsql-hackers@postgresql.org>
Sent: Friday, December 01, 2000 11:02 AM
Subject: Re: [HACKERS] beta testing version

On Fri, Dec 01, 2000 at 11:48:23AM -0800, Don Baccus wrote:

At 11:09 AM 12/1/00 -0800, Nathan Myers wrote:

On Fri, Dec 01, 2000 at 10:01:15AM +0100, Zeugswetter Andreas SB wrote:

If you need to restore from offsite backup you loose transactions
unless you transfer the WAL synchronously with every commit.

Currently the only way to avoid losing those transactions is by
replicating transactions at the application layer. That is, the
application talks to two different database instances, and enters
transactions into both. That's pretty hard to retrofit into an
existing application, so you'd really rather have replication in
the database. Of course, that's something PostgreSQL, Inc. is also
working on.

Recovery alone isn't quite that difficult. You don't need to instantiate
your database instance until you need to apply the archived transactions,
i.e. after catastrophic failure destroys your db server.

True, it's sufficient for the application just to log the text of
its updating transactions off-site. Then, to recover, instantiate
a database from a backup and have the application re-run its
transactions.

You need to do two things:

(Remember, we're talking about what you could do *now*, with 7.1.
Presumably with 7.2 other options will open.)

1. Transmit a consistent (known-state) snapshot of the database offsite.

2. Synchronously tranfer the WAL as part of every commit (question, do
wait to log a "commit" locally until after the remote site acks that
it got the WAL?)

Then you take a new machine, build a database out of the snapshot, and
apply the archived redo logs and off you go. If you get tired of saving
oodles of redo archives, you make a new snapshot and accumulate the
WAL from that point forward.

I don't know of any way to synchronously transfer the WAL, currently.

Anyway, I would expect doing it to interfere seriously with performance.
The "wait to log a 'commit' locally until after the remote site acks that
it got the WAL" is (akin to) the familiar two-phase commit.

Nathan Myers
ncm@zembu.com

At 12:56 PM 12/1/00 -0800, Nathan Myers wrote:

(Remember, we're talking about what you could do *now*, with 7.1.
Presumably with 7.2 other options will open.)

Maybe *you* are :) Seriously, I'm thinking out loud about future
possibilities. Putting a lot of work into building up a temporary
solution on top of 7.1 doesn't make a lot of sense, anyone wanting
to work on such things ought to think about 7.2, which presumably will
beta sometime mid-2001 or so???

And I don't think there are 7.1 hacks that are simple ... could be
wrong, though.

I don't know of any way to synchronously transfer the WAL, currently.

Nope.

Anyway, I would expect doing it to interfere seriously with performance.

Yep. Anyone here have experience with replication and Oracle or others?
I've heard from one source that setting it up reliabily in Oracle and
getting the switch from the dead to the backup server working properly was
something of a DBA nightmare, but that's true of just about anything in
Oracle. Once it was up, it worked reliably, though (also typical
of Oracle).

The "wait to log a 'commit' locally until after the remote site acks that
it got the WAL" is (akin to) the familiar two-phase commit.

Right.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

Don Baccus writes:

Exactly what is PostgreSQL, Inc doing in this area?

Good question... See http://www.erserver.com/.

I've not seen discussions about it here, and the two of the three most
active developers (Jan and Tom) work for Great Bridge, not PostgreSQL,
Inc...

Vadim Mikheev and Thomas Lockhart work for PostgreSQL, Inc., at least in
some form or another. Which *might* be construed as a reason for their
perceived inactivity.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

From: "Nathan Myers" <ncm@zembu.com>

On Thu, Nov 30, 2000 at 07:02:01PM -0400, The Hermit Hacker wrote:

[snip]

The logging in 7.1 protects transactions against many sources of
database crash, but not necessarily against OS crash, and certainly
not against power failure. (You might get lucky, or you might just
think you were lucky.) This is the same as for most databases; an
embedded database that talks directly to the hardware might be able
to do better.

If PG had a type of tree based logging filesystem, that it self handles,
wouldn't that be almost perfectly safe? I mean that you might lose some data
in an transaction, but the client never gets an OK anyways...
Like a combination of raw block io and tux2 like fs.
Doesn't Oracle do it's own block io, no?

Magnus

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
PGP Key: http://www.genline.nu/mag_pgp.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

At 05:42 PM 12/2/00 +0100, Peter Eisentraut wrote:

Don Baccus writes:

Exactly what is PostgreSQL, Inc doing in this area?

Good question... See http://www.erserver.com/.

"Advanced Replication and Distributed Information capabilities are also under development to meet specific
business and competitive requirements for both PostgreSQL, Inc. and clients. Several of these enhanced
PostgreSQL, Inc. developments may remain proprietary for up to 24 months, with availability limited to
clients and partners, in order to assist us in recovering development costs and continue to provide funding
for our other Open Source contributions. "

Boy, I can just imagine the uproar this statement will cause on Slashdot when
the world finds out about it.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Sat, Dec 02, 2000 at 11:31:37AM -0800, Don Baccus wrote:

At 05:42 PM 12/2/00 +0100, Peter Eisentraut wrote:

Don Baccus writes:

Exactly what is PostgreSQL, Inc doing in this area?

Good question... See http://www.erserver.com/.

<snip>

Boy, I can just imagine the uproar this statement will cause on Slashdot when
the world finds out about it.

That one doesn't worry me us much as this quote from the press release at

http://www.pgsql.com/press/PR_5.html

"We expect to have the source code tested and ready to contribute to
the open source community before the middle of October. Until that time
we are considering requests from a number of development companies and
venture capital groups to join us in this process."

Where's the damn core code? I've seen a number of examples already of
people asking about remote access/replication function, with an eye
toward implementing it, and being told "PostgreSQL, Inc. is working
on that". It's almost Microsoftesque: preannounce future functionality
suppressing the competition.

I realize this is probably just the typical deadline slip that we see
on the public releases of pgsql itself, not a silent retraction of the
promise to release the code (especially since some of the same core
people are involved), but there is a difference: if I absolutely need
something that's only in CVS right now, I can bite the bullet and use
a snapshot server. With erserver, I'm stuck sitting on my hands, with a
promise of future functionality. Well, not really sitting on my hands:
working on other tasks, with the assumption that erserver will be there
soon. I'd rather not roll my own in an incompatable way, and have to
port or redo the custom parts.

So, now I'm going into a couple critical, funding decision making
meetings in the next few weeks. I was planning on being able to promise
certain systems with concrete knowledge of what I will and won't be
able to provide, and how much custom coding will be needed. Now, If the
schedsule slips much more, I won't. It's even possible that the erserver's
implementation won't fit my needs at all, and I'll be back rolling my own.

I realize this sounds a bit ungrateful: they're giving away the code,
after all, and potentially saving my a lot of work.

It's just the contrast between the really open work on the core server,
and the lack of a peep when the promised deadlines have rolled past that
gets under my skin.

I'd be really happy with someone reiterating the commitment to an
open release, and letting us all know how badly the schedule has
slipped. Remember, we're all here to help! Get everyone stomping bugs
in code you're going to release soon anyway, and concentrate on the
quasi-propriatary extensions.

Ross

On Sat, 2 Dec 2000, Don Baccus wrote:

...

Will Great Bridge step to the plate and fund a truly open source alternative,
leaving us with a potential code fork? If IB gets its political problems
under control and developers rally around it, two years is going to be a
long time to just sit back and wait for PG, Inc to release eRServer.

I doubt that. There is an IB (Interbase) replication option today, but
you must purchase it. That isn't so bad actually. PostgreSQL looks to be
going that way too: base functionality is open source, periphial
companies make money selling extensions.

Besides simple master-slave replication is old news anyhow, and not
terribly useful. Products like FrontBase (www.frontbase.com) have full
shared-nothing cluster support too (FrontBase is commerical). Clustering
is a much better solution for redundancy purposes that replication.

Tom

At 03:51 PM 12/2/00 -0600, Ross J. Reedstrom wrote:

"We expect to have the source code tested and ready to contribute to
the open source community before the middle of October. Until that time
we are considering requests from a number of development companies and
venture capital groups to join us in this process."

Where's the damn core code? I've seen a number of examples already of
people asking about remote access/replication function, with an eye
toward implementing it, and being told "PostgreSQL, Inc. is working
on that". It's almost Microsoftesque: preannounce future functionality
suppressing the competition.

Well, this is just all 'round a bad precedent and an unwelcome path
for PostgreSQL, Inc to embark upon.

They've also embarked on one fully proprietary product (built on PG),
which means they're not an Open Source company, just a sometimes Open
Source company.

It's a bit ironic to learn about this on the same day I learned that
Solaris 8 is being made available in source form. Sun's slowly "getting
it" and moving glacially towards Open Source, while PostgreSQL, Inc.
seems to be drifting in the opposite direction.

if I absolutely need
something that's only in CVS right now, I can bite the bullet and use
a snapshot server.

This work might be released as Open Source, but it isn't an open development
scenario. The core work's not available for public scrutiny, and the details
of what they're actually up don't appear to be public either.

OK, they're probably funding Vadim's work on WAL, so the idictment's probably
not 100% accurate - but I don't know that.

I'd be really happy with someone reiterating the commitment to an
open release, and letting us all know how badly the schedule has
slipped. Remember, we're all here to help! Get everyone stomping bugs
in code you're going to release soon anyway, and concentrate on the
quasi-propriatary extensions.

Which makes me wonder, is Vadim's time going to be eaten up by working
on these quasi-proprietary extensions that the rest of us won't get
for two years unless we become customers of Postgres, Inc?

Will Great Bridge step to the plate and fund a truly open source alternative,
leaving us with a potential code fork? If IB gets its political problems
under control and developers rally around it, two years is going to be a
long time to just sit back and wait for PG, Inc to release eRServer.

These developments are a major annoyance.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 01:52 PM 12/2/00 -0800, Tom Samplonius wrote:

I doubt that. There is an IB (Interbase) replication option today, but
you must purchase it. That isn't so bad actually. PostgreSQL looks to be
going that way too: base functionality is open source, periphial
companies make money selling extensions.

PostgreSQL, Inc perhaps has that as a game plan. Thus far Great Bridge claims
to be 100% devoted to the Open Source model.

Besides simple master-slave replication is old news anyhow, and not
terribly useful. Products like FrontBase (www.frontbase.com) have full
shared-nothing cluster support too (FrontBase is commerical). Clustering
is a much better solution for redundancy purposes that replication.

I'm not so much concerned about exactly what PG, Inc is planning to offer
as a proprietary piece - I'm purist enough that I worry about what this
signals for their future direction.

If PG, Inc starts doing proprietary chunks, and Great Bridge remains 100%
dedicated to Open Source, I know who I'll want to succeed and prosper.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Sat, Dec 02, 2000 at 03:47:19PM -0800, Adam Haberlach wrote:

Where's the damn core code? I've seen a number of examples already of
people asking about remote access/replication function, with an eye
toward implementing it, and being told "PostgreSQL, Inc. is working
on that". It's almost Microsoftesque: preannounce future functionality
suppressing the competition.

Well, I'll admit that this was getting a little over the top, especially
quoted out of context. ;-)

For What It's Worth: In the three years (has it really been that long?)
that I've been off and on Postgres mailing lists, I've probably seen at
least 100 requests for replication, with about 40 of them mentioning
implementing it themself.

I'm pretty sure that being told "PostgreSQL Inc. is working on that" is
not the only thing stopping it from happening. Most people just aren't up
to making it happen.

Indeed. And it's only been less than a year that that response
has been given. However, it is only in that same timespan that the
functionality and performance of the core server gotten to the point
were replication/remote access is one of immediately fruitful itches to
scratch. We'll see what happens in the future.

Ross

On Sat, Dec 02, 2000 at 03:51:15PM -0600, Ross J. Reedstrom wrote:

On Sat, Dec 02, 2000 at 11:31:37AM -0800, Don Baccus wrote:

At 05:42 PM 12/2/00 +0100, Peter Eisentraut wrote:

Don Baccus writes:

Exactly what is PostgreSQL, Inc doing in this area?

Good question... See http://www.erserver.com/.

<snip>

Boy, I can just imagine the uproar this statement will cause on Slashdot when
the world finds out about it.

That one doesn't worry me us much as this quote from the press release at

http://www.pgsql.com/press/PR_5.html

"We expect to have the source code tested and ready to contribute to
the open source community before the middle of October. Until that time
we are considering requests from a number of development companies and
venture capital groups to join us in this process."

Where's the damn core code? I've seen a number of examples already of
people asking about remote access/replication function, with an eye
toward implementing it, and being told "PostgreSQL, Inc. is working
on that". It's almost Microsoftesque: preannounce future functionality
suppressing the competition.

For What It's Worth: In the three years (has it really been that long?)
that I've been off and on Postgres mailing lists, I've probably seen at
least 100 requests for replication, with about 40 of them mentioning
implementing it themself.

I'm pretty sure that being told "PostgreSQL Inc. is working on that" is
not the only thing stopping it from happening. Most people just aren't up
to making it happen.

--
Adam Haberlach |"California's the big burrito, Texas is the big
adam@newsnipple.com | taco ... and following that theme, Florida is
http://www.newsnipple.com| the big tamale ... and the only tamale that
'88 EX500 | counts any more." -- Dan Rather

PostgreSQL, Inc perhaps has that as a game plan.
I'm not so much concerned about exactly what PG, Inc is planning to offer
as a proprietary piece - I'm purist enough that I worry about what this
signals for their future direction.

Hmm. What has kept replication from happening in the past? It is a big
job and difficult to do correctly. It is entirely my fault that you
haven't seen the demo code released; I've been packaging it to make it a
bit easier to work with.

If PG, Inc starts doing proprietary chunks, and Great Bridge remains 100%
dedicated to Open Source, I know who I'll want to succeed and prosper.

Let me be clear: PostgreSQL Inc. is owned and controlled by people who
have lived the Open Source philosophy, which is not typical of most
companies in business today. We are eager to show how this can be done
on a full time basis, not only as an avocation. And we are eager to do
this as part of the community we have helped to build.

As soon as you find a business model which does not require income, let
me know. The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

- Thomas

At 02:58 AM 12/3/00 +0000, Thomas Lockhart wrote:

PostgreSQL, Inc perhaps has that as a game plan.
I'm not so much concerned about exactly what PG, Inc is planning to offer
as a proprietary piece - I'm purist enough that I worry about what this
signals for their future direction.

Hmm. What has kept replication from happening in the past? It is a big
job and difficult to do correctly.

Presumably what has kept it from happening in the past is that other
things were of much higher priority. Replicating a database on an
engine as unreliable as PG was in earlier incarnations would simply
replicate your problems, for instance.

This statement of yours kinda belittles the work done over the past
few years by volunteers. It also ignores the fact that folks in other
companies do get paid to work on open source software full-time without having
to resort to creating closed source, proprietary products.

It is entirely my fault that you
haven't seen the demo code released; I've been packaging it to make it a
bit easier to work with.

OK, good, this part gets open sourced. Still not an open development model.
Knowing details about what's going on while code's being developed, not to mention
being able to critique decisions, is one of the major benefits of the open
development model.

Let me be clear: PostgreSQL Inc. is owned and controlled by people who
have lived the Open Source philosophy, which is not typical of most
companies in business today. We are eager to show how this can be done
on a full time basis, not only as an avocation.

Building closed source proprietary products helps you live the open source
philosophy on a full-time basis?

...

As soon as you find a business model which does not require income, let
me know.

Red herring, and you know it. The question isn't whether or not your business
generates income, but how it generates income.

Your comment is the classic one tossed out by closed-source, proprietary
software advocates who dismiss open source software out-of-hand.

Couldn't you think of something better, at least? Like ... something
original?

The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

That's a horrible analogy, and I suspect you know it, but at least it is
original.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

This statement of yours kinda belittles the work done over the past
few years by volunteers.

imho it does not, and if somehow you can read that into it then you have
a much different understanding of language than I. I *am* one of those
volunteers, and know that the hundreds of hours I have contributed is
only a small part of the whole.

My discussion on this is over; apologies to others for helping to waste
bandwidth :(

I'll be happy to continue it next over some beers, which is a much more
appropriate setting.

- Thomas

Thomas Lockhart wrote:

PostgreSQL, Inc perhaps has that as a game plan.
I'm not so much concerned about exactly what PG, Inc is planning to offer
as a proprietary piece - I'm purist enough that I worry about what this
signals for their future direction.

Hmm. What has kept replication from happening in the past? It is a big
job and difficult to do correctly.

Well, this has nothing whatsoever to do with open or closed source. Linux
and FreeBSD are much larger, much harder to do correctly, as they are supersets
of thousands of open source projects. Complexity is not relative to licensing.

If PG, Inc starts doing proprietary chunks, and Great Bridge remains 100%
dedicated to Open Source, I know who I'll want to succeed and prosper.

Let me be clear: PostgreSQL Inc. is owned and controlled by people who
have lived the Open Source philosophy, which is not typical of most
companies in business today.

That's one of the reasons why it's worked... open source meant open
contribution, open collaboration, open bug fixing. The price of admission
was doing your own installs, service, support, and giving something back....

PG, I assume, is pretty much the same as most open source projects, massive
amounts of contribution shepherded by one or two individuals.

We are eager to show how this can be done
on a full time basis, not only as an avocation. And we are eager to do
this as part of the community we have helped to build.
As soon as you find a business model which does not require income, let
me know. The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

Well, whether or not a product is open, or closed, has very little
to do with commercial success. Heck, the entire IBM PC spec was open, and
that certainly didn't hurt Dell, Compaq, etc.... the genie coming out
of the bottle _only_ hurt IBM. In this case, however, the genie's been
out for quite a while....

BUT:
People don't buy a product because it's open, they buy it because it offers
significant value above and beyond what they can do *without* paying for
a product. Linus didn't start a new kernel out of some idealistic mantra
of freeing the world, he was broke and wanted a *nix-y OS. Years later,
the product has grown massively. Those who are profiting off of it are
unrelated to the code, to most of the developers.... why is this?

As it is, any company trying to make a closed version of an open source
product has some _massive_ work to do. Manuals. Documentation. Sales.
Branding. Phone support lines. Legal departments/Lawsuit prevention. Figuring
out how to prevent open source from stealing the thunder by duplicating
features. And building a _product_.

Most Open Source projects are not products, they are merely code, and some
horrid documentation, and maybe some support. The companies making money
are not making better code, they are making better _products_....

And I really havn't seen much in the way of full featured products, complete
with printed docs, 24 hour support, tutorials, wizards, templates, a company
to sue if the code causes damage, GUI install, setup, removal, etc. etc. etc.

Want to make money from open source? Well, you have to find, or build,
a _product_. Right now, there are no OS db products that can compare to oh,
an Oracle product, a MSSQL product. There may be superior code, but that
doesn't make a difference in business. Business has very little to do
with building the perfect mousetrap, if nobody can easily use it.

-Bop
--
Brought to you from boop!, the dual boot Linux/Win95 Compaq Presario 1625
laptop, currently running RedHat 6.1. Your bopping may vary.

On Sat, Dec 02, 2000 at 07:32:14PM -0800, Don Baccus wrote:

At 02:58 AM 12/3/00 +0000, Thomas Lockhart wrote:

PostgreSQL, Inc perhaps has that as a game plan.
I'm not so much concerned about exactly what PG, Inc is planning to offer
as a proprietary piece - I'm purist enough that I worry about what this

.
.
.

As soon as you find a business model which does not require income, let
me know.

Red herring, and you know it. The question isn't whether or not your business
generates income, but how it generates income.

So far, Open Source doesn't. The VA Linux IPO made ME some income,
but I'm not sure that was part of their plan...

Your comment is the classic one tossed out by closed-source, proprietary
software advocates who dismiss open source software out-of-hand.

Couldn't you think of something better, at least? Like ... something
original?

The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

That's a horrible analogy, and I suspect you know it, but at least it is
original.

It wasn't an analogy.

In any case, can we create pgsql-politics so we don't have to go over
this issue every three months? Can we create pgsql-benchmarks while we
are at it, to take care of the other thread that keeps popping up?

--
Adam Haberlach |"California's the big burrito, Texas is the big
adam@newsnipple.com | taco ... and following that theme, Florida is
http://www.newsnipple.com| the big tamale ... and the only tamale that
'88 EX500 | counts any more." -- Dan Rather

And I really havn't seen much in the way of full featured products, complete
with printed docs, 24 hour support, tutorials, wizards, templates, a company
to sue if the code causes damage, GUI install, setup, removal, etc. etc. etc.

Mac OS X.

;-)

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

At 09:29 PM 12/2/00 -0800, Adam Haberlach wrote:

Red herring, and you know it. The question isn't whether or not your business
generates income, but how it generates income.

So far, Open Source doesn't. The VA Linux IPO made ME some income,
but I'm not sure that was part of their plan...

VA Linux is a HARDWARE COMPANY. They sell servers. "We've engineered 2U
performance into a 1U box" is their current line.

Dell probably makes more money on their Linux server offerings (I have to
admit that donb.photo.net is running on one of their PowerEdge servers) than
VA Linux does.

If I can show you a HARDWARE COMPANY that is diving on selling MS NT servers,
will you agree that this proves that the closed source and open source models
both must be wrong, because HARDWARE COMPANIES based on each paradigm are
losing money???

The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

That's a horrible analogy, and I suspect you know it, but at least it is
original.

It wasn't an analogy.

Sure it is. Read, damn it. First he makes the statement that a business
based on open source is, by definition, a zero-revenue company then he
raises the spectre of .com companies (how many of them are open source?)
as support for his argument.

OK, it's not an analogy, it's a disassociation with reality. Feel better?

In any case, can we create pgsql-politics so we don't have to go over
this issue every three months?

Maybe you don't care about the open source aspect of this, but as a user
with about 1500 Open Source advocates using my code, I do. If IB comes
forth in a fully Open Source state my user base will insist I switch.

And I will.

And I'll stop telling the world that MySQL sucks, too. Or at least that
they suck worse than the PG world :)

There is risk here. It isn't so much in the fact that PostgreSQL, Inc
is doing a couple of modest closed-source things with the code. After
all, the PG community has long acknowleged that the BSD license would
allow others to co-op the code and commercialize it with no obligations.

It is rather sad to see PG, Inc. take the first step in this direction.

How long until the entire code base gets co-opted?

(Yeah, that's extremist, but seeing PG, Inc. lay down the formal foundation
for such co-opting by taking the first step might well make the potential
reality become real. It certainly puts some of the long-term developers
in no position to argue against such a co-opted snitch of the code).

I have to say I'm feeling pretty silly about raising such an effort to
increase PG awareness in mindshare vs. MySQL. I mean, if PG, Inc's
efforts somehow delineate the hopes and goals of the PG community, I'm
fairly disgusted.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 04:42 AM 12/3/00 +0000, Thomas Lockhart wrote:

This statement of yours kinda belittles the work done over the past
few years by volunteers.

imho it does not,

Sure it does. You in essence are saying that "advanced replication is so
hard that it could only come about if someone were willing to finance a
PROPRIETARY solution. The PG developer group couldn't manage it if
it were done Open Source".

In other words, it is much harder than any of the work done by the
same group of people before they started working on proprietary
versions.

And that the only way to get them doing their best work is to put them
on proprietary, or "semi-proprietary" projects, though 24 months from
now, who's going to care? You've opened the door to IB prominence, not
only shooting PG's open source purity down in flames, but probably PG, Inc's
as well - IF IB can figure out their political problems.

IB, as it stands, is a damned good product in many ways ahead of PG. You're
giving them life by this approach, which is a kind of bizarre businees strategy.

I *am* one of those volunteers

Yes, I well remember you screwing up PG 7.0 just before beta, without bothering
to test your code, and leaving on vacation.

You were irresponsible then, and you're being irresponsible now.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 09:56 PM 12/2/00 -0700, Ron Chmara wrote:
...

And I really havn't seen much in the way of full featured products, complete
with printed docs, 24 hour support, tutorials, wizards, templates, a company
to sue if the code causes damage, GUI install, setup, removal, etc. etc. etc.

Want to make money from open source? Well, you have to find, or build,
a _product_. Right now, there are no OS db products that can compare to oh,
an Oracle product, a MSSQL product. There may be superior code, but that
doesn't make a difference in business. Business has very little to do
with building the perfect mousetrap, if nobody can easily use it.

Which of course is the business model - certainly not a "zero revenue" model
as Thomas arrogantly suggests - which OSS service companies are following.

They provide the cocoon around the code.

I buy RH releases from Fry's. Yes, I could download, but the price is such
that I'd rather just go buy the damned release CDs. I don't begrudge it,
they're providing me a real SERVICE, saving me time, which saves me dollars
in opportunity costs (given my $200/hr customer billing rate). They make
money buy publishing releases, I still get all the sources. We all win.

It is not a bad model.

Question - if this model sucks, then certainly PG, Inc's net revenue last
year was greater than any true open source software company's? I mean, let's
see that slam against the "zero revenue business model" be proven by showing
us some real numbers.

Just what was PG, Inc's net revenue last year, and just how does their mixed
revenue model stack up against the OSS world?

(NOT the .com world, which is in a different business, no matter what Thomas
wants to claim).

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

There is risk here. It isn't so much in the fact that PostgreSQL, Inc
is doing a couple of modest closed-source things with the code. After
all, the PG community has long acknowleged that the BSD license would
allow others to co-op the code and commercialize it with no obligations.

It is rather sad to see PG, Inc. take the first step in this direction.

How long until the entire code base gets co-opted?

I totaly missed your point here. How closing source of ERserver is related
to closing code of PostgreSQL DB server? Let me clear things:

1. ERserver isn't based on WAL. It will work with any version >= 6.5

2. WAL was partially sponsored by my employer, Sectorbase.com,
not by PG, Inc.

Vadim

Don Baccus <dhogaza@pacifier.com> writes:

At 04:42 AM 12/3/00 +0000, Thomas Lockhart wrote:

This statement of yours kinda belittles the work done over the past
few years by volunteers.

imho it does not,

Sure it does. You in essence are saying that "advanced replication is so
hard that it could only come about if someone were willing to finance a
PROPRIETARY solution. The PG developer group couldn't manage it if
it were done Open Source".

<snip>

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

Mr. Baccus,

It is funny how you rant and rave about the importance of opensource
and how Postgresql Inc. making an non-opensource product is bad. Yet I
go to your website which is full of photographs and you make it a big
deal about people should not steal your photographs and how someone
must buy a commercial license to use them. That doesn't sound very
'open-source' to me! Why don't you practice what you preach and allow
redistribution of those photographs?

--
Prasanth Kumar
kumar1@home.com

Don Baccus writes:

How long until the entire code base gets co-opted?

Yeah so what? Nobody's forcing you to use, buy, or pay attention to any
such efforts. The market will determine whether the release model of
PostgreSQL, Inc. appeals to customers. Open source software is a
privilege, and nobody has the right to call someone "irresponsible"
because they want to get paid for their work and don't choose to give away
their code.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Ron Chmara wrote:

As it is, any company trying to make a closed version of an open source
product has some _massive_ work to do. Manuals. Documentation. Sales.
Branding. Phone support lines. Legal departments/Lawsuit prevention. Figuring
out how to prevent open source from stealing the thunder by duplicating
features. And building a _product_.

Most Open Source projects are not products, they are merely code, and some
horrid documentation, and maybe some support. The companies making money
are not making better code, they are making better _products_....

And I really havn't seen much in the way of full featured products, complete
with printed docs, 24 hour support, tutorials, wizards, templates, a company
to sue if the code causes damage, GUI install, setup, removal, etc. etc. etc.

This kind of stuff is more along the lines of what Great Bridge is doing. In about
a week, we'll be releasing a GB-branded release of 7.0.3 - including printed
manuals (much of which is new), a GUI installer (which is open source), support
packages including fully-staffed 24/7. Details to follow soon on pgsql-announce.

I don't want to speak for Pgsql Inc., but it seems to me that they are pursuing a
slightly different business model than us - more focused on providing custom
development around the base PostgreSQL software. And that's a great way to get
more people using PostgreSQL. Some of what they create for their customers may be
open source, some not. It's certainly their decision - and it's a perfectly
justifiable business model, followed by open source companies such as Covalent
(Apache), Zend (PHP), and TurboLinux. I don't think it's productive or appropriate
to beat up on Pgsql Inc for developing bolt-on products in a different way -
particularly with Vadim's clarification that the bolt-ons don't require anything
special in the open source backend.

Our own business model is, as I indicated, different. We got a substantial
investment from our parent company, whose chairman sat on the Red Hat board for
three years, and a mandate to create a *big* company that could provide the
infrastructure (human and technical) to enable PostgreSQL to go up against the
proprietary players like Oracle and Microsoft. A fully-staffed 24/7 data center
isn't cheap, and our services won't be either. But it's a different type of
business - we're providing the benefits of the open source development model to a
group of customers that might not otherwise get involved, precisely because they
demand to see a company of Great Bridge's heft behind a product before they buy.

I think PostgreSQL and other open source projects are big enough for lots of
different companies, with lots of different types of business models. Indeed, from
what I've seen of Pgsql Inc (and I hope I haven't mischaracterized them), our
business models are highly complementary. At Great Bridge, we hope and expect that
other companies that "get it" will get more involved with PostgreSQL - that can
only add to the strength of the project.

Regards,
Ned
--
----------------------------------------------------
Ned Lilly e: ned@greatbridge.com
Vice President w: www.greatbridge.com
Evangelism / Hacker Relations v: 757.233.5523
Great Bridge, LLC f: 757.233.5555

Branding. Phone support lines. Legal departments/Lawsuit prevention.

Figuring

out how to prevent open source from stealing the thunder by duplicating

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

features. And building a _product_.

Oops. You didn't really mean that, did you? Could it be that there are some
people out there thinking "let them free software fools do the hard initial
work, once things are working nicely, we take over, add a few "secret"
ingredients, and voila - the commercial product has been created?

After reading the statement above I believe that surely most of the honest
developers involved in postgres would wish they had chosen GPL as licensing
scheme.

I agree that most of the work is always done by a few. I also agree that it
would be nice if they could get some financial reward for it. But no dirty
tricks please. Do not betray the base. Otherwise, the broad developer base
will be gone before you even can say "freesoftware".

I, for my part, have learned another lesson today. I was just about to give
in with the licensing scheme in our project to allow the GPL incompatible
OpenSSL to be used. After reading the above now I know it is worth the extra
effort to "roll our own" or wait for another GPL'd solution rather than
sacrificing the unique protection the GPL gives us.

Horst
coordinator gnumed project

How long until the entire code base gets co-opted?

Yeah so what? Nobody's forcing you to use, buy, or pay attention to any
such efforts. The market will determine whether the release model of
PostgreSQL, Inc. appeals to customers. Open source software is a
privilege, and nobody has the right to call someone "irresponsible"
because they want to get paid for their work and don't choose to give away
their code.

Just bear in mind that although a few developers always deliver outstanding
performance in any project, those open source projects have usually seen a
huge broad developer base. Hundreds of people putting their effort into the
project. These people never ask for a cent, never even dream of some
commercial benefit. They do it for the sake of creating something good,
being part of something great.

Especially in the case of Postgres the "product" has a long heritage, and
the most active people today are not neccessarily the ones who have put in
most "total" effort (AFAIK, I might be wrong here). Anyway, Postgres would
not be where it is today without the hundreds of small cooperators &
testers. Lock them out from the source code - even if it is only a side
branch, and Postgres will die (well, at least it would die for our project)

Open source is not a mere marketing model. It is a philosophy. It is about
essential freedom, about human progress, about freedom of speech and
thought. It is about sharing and caring. Those who don't understand this,
should please stick to their ropes and develop closed source from the
beginning and not try to fool the free software community.

Horst

Thomas Lockhart wrote:

As soon as you find a business model which does not require income, let
me know. The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

While I have not contributed anything to Postgres yet, I have
contributed to other environments. The prospect that I could create a
piece of code, spend weeks/years of my own time on something and some
entity can come along, take what I've written and create a product which
is better for it, and then not share back is offensive. Under GPL it is
illegal. (Postgres should try to move to GPL)

I am working on a full-text search engine for Postgres. A really fast
one, something better than anything else out there. It combines the
power and scalability of a web search engine, with the data-mining
capabilities of SQL.

If I write this extension to Postgres, and release it, is it right that
a business can come along, add a few things here and there and introduce
a new closed source product on what I have written? That is certainly
not what I intend. My intention was to honor the people before me for
providing the rich environment which is Postgres. I have made real money
using Postgres in a work environment. The time I would give back more
than covers MSSQL/Oracle licenses.

Open source is a social agreement, not a business model. If you break
the social agreement for a business model, the business model will fail
because the society which fundamentally created the product you wish to
sell will crumble from mistrust (or shun you). In short, it is wrong to
sell the work of others without proper compensation and the full
agreement of everyone that has contributed. If you don't get that, get
out of the open source market now.

That said, there is a long standing business model which is 100%
compatible with Open Source and it is of the lowly 'VAR.' You do not
think for one minute that an Oracle VAR would dare to add features to
Oracle and make their own SQL do you?

As a PostgreSQL "VAR" you are in a better position that any other VAR.
You get to partner in the code development process. (You couldn't ask
Oracle to add a feature and expect to keep it to yourself, could you?)

I know this is a borderline rant, and I am sorry, but I think it is very
important that the integrity of open source be preserved at 100% because
it is a very slippery slope, and we are all surrounded by the temptation
cheat the spirit of open source "just a little" for short term gain.

--
http://www.mohawksoft.com

At 11:00 PM 12/2/00 -0800, Vadim Mikheev wrote:

There is risk here. It isn't so much in the fact that PostgreSQL, Inc
is doing a couple of modest closed-source things with the code. After
all, the PG community has long acknowleged that the BSD license would
allow others to co-op the code and commercialize it with no obligations.

It is rather sad to see PG, Inc. take the first step in this direction.

How long until the entire code base gets co-opted?

I totaly missed your point here. How closing source of ERserver is related
to closing code of PostgreSQL DB server? Let me clear things:

(not based on WAL)

That's wasn't clear from the blurb.

Still, this notion that PG, Inc will start producing closed-source products
poisons the well. It strengthens FUD arguments of the "open source can't
provide enterprise solutions" variety. "Look, even PostgreSQL, Inc realizes
that you must follow a close sourced model in order to provide tools for
the corporate world."

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 01:06 PM 12/3/00 +0100, Peter Eisentraut wrote:

Open source software is a
privilege,

I admit that I don't subscribe to Stallman's "source to software is a
right" argument. That's far off my reality map.

and nobody has the right to call someone "irresponsible"
because they want to get paid for their work and don't choose to give away
their code.

However, I do have the right to make such statements, just as you have the
right to disagree. It's called the first amendment in my country.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

I totaly missed your point here. How closing source of ERserver is related
to closing code of PostgreSQL DB server? Let me clear things:

(not based on WAL)

That's wasn't clear from the blurb.

Still, this notion that PG, Inc will start producing closed-source products
poisons the well. It strengthens FUD arguments of the "open source can't
provide enterprise solutions" variety. "Look, even PostgreSQL, Inc realizes
that you must follow a close sourced model
in order to provide tools for the corporate world."

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Did you miss Thomas' answer? Wasn't it clear that the order is to provide
income?

Vadim

I think this trend is MUCH bigger than what Postgres, Inc. is doing... its
happening all over
the comminity. Heck take a look around... Jabber, Postgres, Red Hat, SuSe,
Storm etc. etc.
these companies are making good money off a business plan that was basically
"hey, lets take some
of that open source and make a real product out of it...". As long as they
dribble releases into
the community, they're not in violation... Its not a bad business model if
you think about it, if you
can take a product that is good (great as in PG) and add value, sell it and
make money, why not?
Hell, you didn't have to spend the gazillion R&D dollars on the initial
design and implementation,
your basically reaping the rewards off of the work of other people.

Are you ready for hundreds upon hundreds of little projects turning into
"startup" companies?
It was bound to happen. Why? because money is involved, plain and simple.

Maybe its a natural progression of this stuff, who knows, I just know that
I've been around
the block a couple times, been in the industry too long to know that the
minority voice never
gets the prize... we usually set the trend and pay for it in the end...
fatalistic? maybe. But not
far from the truth...

Sorry to be a downer... The Red Sox didn't get Mussina....

----- Original Message -----
From: "Don Baccus" <dhogaza@pacifier.com>
To: "Ross J. Reedstrom" <reedstrm@rice.edu>
Cc: "Peter Eisentraut" <peter_e@gmx.net>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Saturday, December 02, 2000 5:11 PM
Subject: Re: [HACKERS] beta testing version

At 03:51 PM 12/2/00 -0600, Ross J. Reedstrom wrote:

"We expect to have the source code tested and ready to contribute to
the open source community before the middle of October. Until that time
we are considering requests from a number of development companies and
venture capital groups to join us in this process."

Where's the damn core code? I've seen a number of examples already of
people asking about remote access/replication function, with an eye
toward implementing it, and being told "PostgreSQL, Inc. is working
on that". It's almost Microsoftesque: preannounce future functionality
suppressing the competition.

Well, this is just all 'round a bad precedent and an unwelcome path
for PostgreSQL, Inc to embark upon.

They've also embarked on one fully proprietary product (built on PG),
which means they're not an Open Source company, just a sometimes Open
Source company.

It's a bit ironic to learn about this on the same day I learned that
Solaris 8 is being made available in source form. Sun's slowly "getting
it" and moving glacially towards Open Source, while PostgreSQL, Inc.
seems to be drifting in the opposite direction.

if I absolutely need
something that's only in CVS right now, I can bite the bullet and use
a snapshot server.

This work might be released as Open Source, but it isn't an open

development

scenario. The core work's not available for public scrutiny, and the

details

of what they're actually up don't appear to be public either.

OK, they're probably funding Vadim's work on WAL, so the idictment's

probably

not 100% accurate - but I don't know that.

I'd be really happy with someone reiterating the commitment to an
open release, and letting us all know how badly the schedule has
slipped. Remember, we're all here to help! Get everyone stomping bugs
in code you're going to release soon anyway, and concentrate on the
quasi-propriatary extensions.

Which makes me wonder, is Vadim's time going to be eaten up by working
on these quasi-proprietary extensions that the rest of us won't get
for two years unless we become customers of Postgres, Inc?

Will Great Bridge step to the plate and fund a truly open source

alternative,

Peter Eisentraut wrote:

mlw writes:

There are hundreds (thousands?) of people that have contributed to the
development of Postgres, either directly with code, or beta testing,
with the assumption that they are benefiting a community. Many would
probably not have done so if they had suspected that what they do is
used in a product that excludes them.

With the BSD license it has always been clear that this would be possible,
and for as long as I've been around the core/active developers have
frequently reiterated that this is a desirable aspect and in fact
encouraged. If you don't like that, then you should have read the license
before using the product.

I have said before, open source is a social contract, not a business
model.

Well, you're free to take the PostgreSQL source and start your own "social
contract" project; but we don't do that around here.

And you don't feel that this is a misappropriation of a public trust? I
feel shame for you.

--
http://www.mohawksoft.com

On Sat, 2 Dec 2000, Adam Haberlach wrote:

In any case, can we create pgsql-politics so we don't have to go over
this issue every three months? Can we create pgsql-benchmarks while we
are at it, to take care of the other thread that keeps popping up?

no skin off my back:

pgsql-advocacy
pgsql-chat
pgsql-benchmarks

-advocacy/-chat are pretty much the same concept ...

On Sat, 2 Dec 2000, Don Baccus wrote:

I *am* one of those volunteers

Yes, I well remember you screwing up PG 7.0 just before beta, without bothering
to test your code, and leaving on vacation.

You were irresponsible then, and you're being irresponsible now.

Okay, so let me get this one straight ... it was irresponsible for him to
put code in that was broken the last time, but it wouldn't be
irresponsible for us to release code that we don't feel is ready this
time? *raised eyebrow*

Just want to get this straight, as it kinda sounds hypocritical to me, but
want to make sure that I understand before I fully arrive at that
conclusion ... :)

Don Baccus wrote:

At 04:42 AM 12/3/00 +0000, Thomas Lockhart wrote:

This statement of yours kinda belittles the work done over the past
few years by volunteers.

imho it does not,

Sure it does. You in essence are saying that "advanced replication is so
hard that it could only come about if someone were willing to finance a
PROPRIETARY solution. The PG developer group couldn't manage it if
it were done Open Source".

In other words, it is much harder than any of the work done by the
same group of people before they started working on proprietary
versions.

And that the only way to get them doing their best work is to put them
on proprietary, or "semi-proprietary" projects, though 24 months from
now, who's going to care? You've opened the door to IB prominence, not
only shooting PG's open source purity down in flames, but probably PG, Inc's
as well - IF IB can figure out their political problems.

IB, as it stands, is a damned good product in many ways ahead of PG. You're
giving them life by this approach, which is a kind of bizarre businees strategy.

You (and others ;) may also be interested in SAPDB (SAP's version of
Adabas),
that is soon to be released under GPL. It is already downloadable for
free use
from www.sapdb.org

-------------
Hannu

On Mon, 4 Dec 2000, Horst Herb wrote:

Branding. Phone support lines. Legal departments/Lawsuit prevention.

Figuring

out how to prevent open source from stealing the thunder by duplicating

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

features. And building a _product_.

Oops. You didn't really mean that, did you? Could it be that there are some
people out there thinking "let them free software fools do the hard initial
work, once things are working nicely, we take over, add a few "secret"
ingredients, and voila - the commercial product has been created?

After reading the statement above I believe that surely most of the
honest developers involved in postgres would wish they had chosen GPL
as licensing scheme.

I agree that most of the work is always done by a few. I also agree
that it would be nice if they could get some financial reward for it.
But no dirty tricks please. Do not betray the base. Otherwise, the
broad developer base will be gone before you even can say
"freesoftware".

I, for my part, have learned another lesson today. I was just about to
give in with the licensing scheme in our project to allow the GPL
incompatible OpenSSL to be used. After reading the above now I know it
is worth the extra effort to "roll our own" or wait for another GPL'd
solution rather than sacrificing the unique protection the GPL gives
us.

to this day, this still cracks me up ... if a BSD licensed OSS project
somehow gets its code base "closed", that closing can only affect the code
base from its closing on forward ... on that day, there is *nothing*
stopping from the OSS community from taking the code base from teh second
before it was closed and running with it ...

you get no more, and no less, protection under either license.

the "protection" that GPL provides is that it prevents someone from taking
the code, making proprietary modications to it and branding it as their
own for release ... cause under GPL, they would have to release the source
code for the modifications ...

PgSQL, Inc hasn't done anything so far but develop third party
*applications* over top of PgSQL, with plans to release them at various
stages as the clients we are developing them for permit ... as well as
provided consulting to clients looking at moving towards PgSQL and
requiring help with migrations ...

We aren't going to release something that is half-assed and buggy ... the
whole erServer stuff right now is *totally* external to the PgSQL server,
and, as such, is a third-party application, not a proprietary extension
like Don wants to make it out to be ...

"Gary MacDougall" <gary@freeportweb.com> writes:

I think this trend is MUCH bigger than what Postgres, Inc. is
doing... its happening all over the comminity. Heck take a look
around... Jabber, Postgres, Red Hat, SuSe, Storm etc. etc. these
companies are making good money off a business plan that was basically
"hey, lets take some of that open source and make a real product out
of it...".

I doubt many of these "make good money". We're almost breaking even,
which is probably the best among these.

Note also that some companies contribute engineering resources into
core free software components, like gcc, gdb, the linux kernel, glibc,
gnome, gtk+, rpm, apache, XFree, KDE - AFAIK, Red Hat and SuSE are by
far the two doing this the most.

--
Trond Eivind Glomsr�d
Red Hat, Inc.

Adam Haberlach wrote:

In any case, can we create pgsql-politics so we don't have to go over
this issue every three months? Can we create pgsql-benchmarks while we
are at it, to take care of the other thread that keeps popping up?

pgsql-yawn, where any of them can happen as often and long as
they want.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

mlw wrote:

Thomas Lockhart wrote:

As soon as you find a business model which does not require income, let
me know. The .com'ers are trying it at the moment, and there seems to be
a few flaws... ;)

While I have not contributed anything to Postgres yet, I have
contributed to other environments. The prospect that I could create a
piece of code, spend weeks/years of my own time on something and some
entity can come along, take what I've written and create a product which
is better for it, and then not share back is offensive. Under GPL it is
illegal. (Postgres should try to move to GPL)

I think that forbidding anyone else from profiting from your work is
also
somewhat obscene ;)

The whole idea of open source is that in open ideas mature faster, bugs
are

I am working on a full-text search engine for Postgres. A really fast
one, something better than anything else out there.

Is'nt everybody ;)

It combines the power and scalability of a web search engine, with
the data-mining capabilities of SQL.

Are you doing it in a fully open-source fashion or just planning to
release
it as OS "when it somewhat works" ?

If I write this extension to Postgres, and release it, is it right that
a business can come along, add a few things here and there and introduce
a new closed source product on what I have written? That is certainly
not what I intend.

If your intention is to later cash in on proprietary uses of your code
you should of course use GPL.

My intention was to honor the people before me for
providing the rich environment which is Postgres. I have made real money
using Postgres in a work environment. The time I would give back more
than covers MSSQL/Oracle licenses.

Open source is a social agreement, not a business model.

Not one but many (and btw. incompatible) social agreements.

If you break the social agreement for a business model,

You are free to put your additions under GPL, it is just a tradition in
PG
community not to contaminate the core with anything less free than BSD
(and yes,
forcing your idea of freedom on other people qualifies as "less free" ;)

the business model will fail
because the society which fundamentally created the product you wish to
sell will crumble from mistrust (or shun you). In short, it is wrong to
sell the work of others without proper compensation and the full
agreement of everyone that has contributed. If you don't get that, get
out of the open source market now.

SO now a social contract is a market ? I _am_ confused.

That said, there is a long standing business model which is 100%
compatible with Open Source and it is of the lowly 'VAR.' You do not
think for one minute that an Oracle VAR would dare to add features to
Oracle and make their own SQL do you?

But if Oracle were released under BSD license, it might benefit both the
VAR and the customer to do so under some circumstances.

As a PostgreSQL "VAR" you are in a better position that any other VAR.
You get to partner in the code development process. (You couldn't ask
Oracle to add a feature and expect to keep it to yourself, could you?)

You could ask another VAR to do that if you yourself are incapable/don't
have time, etc.

And of course I can keep it to myself even if done by Oracle.
What I can't do is forbid others from having it too .

I know this is a borderline rant, and I am sorry, but I think it is very
important that the integrity of open source be preserved at 100% because
it is a very slippery slope, and we are all surrounded by the temptation
cheat the spirit of open source "just a little" for short term gain.

Do you mean that anyone who has contributed to an opensource project
should
be forbidden from doing any closed-source development ?

-----------
Hannu

The Hermit Hacker wrote:

On Sat, 2 Dec 2000, Don Baccus wrote:

I *am* one of those volunteers

Yes, I well remember you screwing up PG 7.0 just before beta, without bothering
to test your code, and leaving on vacation.

You were irresponsible then, and you're being irresponsible now.

Okay, so let me get this one straight ... it was irresponsible for him to
put code in that was broken the last time, but it wouldn't be
irresponsible for us to release code that we don't feel is ready this
time? *raised eyebrow*

Just want to get this straight, as it kinda sounds hypocritical to me, but
want to make sure that I understand before I fully arrive at that
conclusion ... :)

IIRC, this thread woke up on someone complaining about PostgreSQl inc
promising
to release some code for replication in mid-october and asking for
confirmation
that this is just a schedule slip and that the project is still going on
and
going to be released as open source.

What seems to be the answer is: "NO, we will keep the replication code
proprietary".

I have not seen this answer myself, but i've got this impression from
the contents
of the whole discussion.

Do you know if this is the case ?

-----------
Hannu

Hannu Krosing wrote:

I know this is a borderline rant, and I am sorry, but I think it is very
important that the integrity of open source be preserved at 100% because
it is a very slippery slope, and we are all surrounded by the temptation
cheat the spirit of open source "just a little" for short term gain.

Do you mean that anyone who has contributed to an opensource project
should be forbidden from doing any closed-source development ?

No, not at all. At least for me, if I write code which is dependent on
the open source work of others, then hell yes, that work should also be
open source. That, to me, is the difference between right and wrong.

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

I honestly feel that it is wrong to take what others have shared and use
it for the basis of something you will not share, and I can't understand
how anyone could think differently.

--
http://www.mohawksoft.com

mlw wrote:

Hannu Krosing wrote:

I know this is a borderline rant, and I am sorry, but I think it is very
important that the integrity of open source be preserved at 100% because
it is a very slippery slope, and we are all surrounded by the temptation
cheat the spirit of open source "just a little" for short term gain.

Do you mean that anyone who has contributed to an opensource project
should be forbidden from doing any closed-source development ?

No, not at all. At least for me, if I write code which is dependent on
the open source work of others, then hell yes, that work should also be
open source. That, to me, is the difference between right and wrong.

That may be so, that the world as a whole is not that far yet. If
open-source
is going to prevail (which I believe it will do), it is not because it
is
"right", but because it is a more efficient way of producing quality
software.

I honestly feel that it is wrong to take what others have shared and use
it for the basis of something you will not share, and I can't understand
how anyone could think differently.

There can be many many reasons you would need to also write
closed-source code.
BSD license gives you that freedom (GPL does not).

By distributing your code under BSD license you acnowledge that the
world is
not perfect. This is not the way of a true revolutionary ;)

Don't let that scare you away from contributing to PostgreSQL though,
You could
always contribute and keep your code under different license,
GPL,LGPL,MPL, ...
It would probably not be integrated in the core, but would very likely
be kept
in contrib.

----------
Hannu

No offense Trond, if you were in on the Red Hat IPO from the start,
you'd have to say those people made "good money". Bad market
or good market, those "friends of Red Hat" made some serious coin.

Let me clarify, I'm not against this process (and making money), I just
think there is an issue with OSL that will start to catch up with itself
pretty soon.

g.

----- Original Message -----
From: "Trond Eivind Glomsr�d" <teg@redhat.com>
To: "PostgreSQL Development" <pgsql-hackers@postgresql.org>
Sent: Sunday, December 03, 2000 4:24 PM
Subject: Re: [HACKERS] beta testing version

"Gary MacDougall" <gary@freeportweb.com> writes:

I think this trend is MUCH bigger than what Postgres, Inc. is
doing... its happening all over the comminity. Heck take a look
around... Jabber, Postgres, Red Hat, SuSe, Storm etc. etc. these
companies are making good money off a business plan that was basically
"hey, lets take some of that open source and make a real product out
of it...".

I doubt many of these "make good money". We're almost breaking even,
which is probably the best among these.

Note also that some companies contribute engineering resources into
core free software components, like gcc, gdb, the linux kernel, glibc,
gnome, gtk+, rpm, apache, XFree, KDE - AFAIK, Red Hat and SuSE are by
far the two doing this the most.

--
Trond Eivind Glomsr�d
Red Hat, Inc.

"Gary MacDougall" <gary@freeportweb.com> writes:

No offense Trond, if you were in on the Red Hat IPO from the start,
you'd have to say those people made "good money".

I'm talking about the business as such, not the IPO where the price
went stratospheric (we were priced like we were earning 1 or 2 billion
dollars year, which was kindof weird).

--
Trond Eivind Glomsr�d
Red Hat, Inc.

No, not at all. At least for me, if I write code which is dependent on
the open source work of others, then hell yes, that work should also be
open source. That, to me, is the difference between right and wrong.

Actually, your not legally bound to anything if you write "new" additional
code, even if its dependant on something. You could consider it
"propietary"
and charge for it. There a tons of these things going on right now.

Having dependancy on an open source product/code/functionality does not
make one bound to make thier code "open source".

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

Thats a given.

I honestly feel that it is wrong to take what others have shared and use
it for the basis of something you will not share, and I can't understand
how anyone could think differently.

The issue isn't "fairness", the issue really is really trust. And from what
I'm
seeing, like anything else in life, if you rely solely on trust when money
is
involved, the system will fail--eventually.

sad... isn't it?

Gary MacDougall wrote:

No, not at all. At least for me, if I write code which is dependent on
the open source work of others, then hell yes, that work should also be
open source. That, to me, is the difference between right and wrong.

Actually, your not legally bound to anything if you write "new" additional
code, even if its dependant on something. You could consider it
"propietary"
and charge for it. There a tons of these things going on right now.

Having dependancy on an open source product/code/functionality does not
make one bound to make thier code "open source".

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

Thats a given.

I honestly feel that it is wrong to take what others have shared and use
it for the basis of something you will not share, and I can't understand
how anyone could think differently.

The issue isn't "fairness", the issue really is really trust. And from what
I'm
seeing, like anything else in life, if you rely solely on trust when money
is
involved, the system will fail--eventually.

sad... isn't it?

That's why, as bad as it is, GPL is the best answer.

--
http://www.mohawksoft.com

On Sun, Dec 03, 2000 at 05:17:36PM -0500, mlw wrote:

... if I write code which is dependent on
the open source work of others, then hell yes, that work should also be
open source. That, to me, is the difference between right and wrong.

This is short and I will say no more:

The entire social contract around PostgreSQL is written down in the
license. Those who have contributed to the project (are presumed to)
have read it and agreed to it before submitting their changes. Some
people have contributed intending someday to fold the resulting code
base into their proprietary product, and carefully checked to ensure
the license would allow it. Nobody has any legal or moral right to
impose extra use restrictions, on their own code or (especially!) on
anybody else's.

If you would like to place additional restrictions on your own
contributions, you can:

1. Work on other projects. (Adabas will soon be GPL, but you can
start now. Others are coming, too.) There's always plenty of
work to be done on Free Software.

2. Fork the source base, add your code, and release the whole thing
under GPL. You can even fold in changes from the original project,
later. (Don't expect everybody to get along, afterward.) A less
drastic alternative is to release GPL'd patches.

3. Grin and bear it. Greed is a sin, but so is envy.

Flame wars about licensing mainly distract people from writing code.
How would *you* like the time spent?

Nathan Myers
ncm@zembu.com

mlw wrote: [heavily edited]

No, not at all. At least for me, if I write code which is dependent on
the open source work of others, then hell yes, that work should also be
open source. That, to me, is the difference between right and wrong.
I honestly feel that it is wrong to take what others have shared and use
it for the basis of something you will not share, and I can't understand
how anyone could think differently.

You're missing the point almost completely. We've been around on this
GPL-vs-BSD discussion many many many times before, and the discussion
always ends up at the same place: we aren't changing the license.

The two key reasons (IMHO) are:

1. The original code base is BSD. We do not have the right to
unilaterally relabel that code as GPL. Maybe we could try to say that
all additions/changes after a certain date are GPL, but that'd become a
hopeless mess very shortly; how would you keep track of what was which?
Not to mention the fact that a mixed-license project would not satisfy
GPL partisans anyway.

2. Since Postgres is a database, and the vast majority of uses for
databases are business-related, we have to have a license that
businesses will feel comfortable with. One aspect of that comfort is
that they be able to do things like building proprietary applications
atop the database. If we take a purist GPL approach, we'll just drive
away a lot of potential users and contributors. (I for one wouldn't be
here today, most likely, if Postgres had been GPL --- my then company
would not have gotten involved with it.)

I have nothing against GPL; it's appropriate for some things. But
it's not appropriate for *this* project, because of history and subject
matter. We've done just fine with the BSD license and I do not see a
reason to think that GPL would be an improvement.

regards, tom lane

At 5:17 PM -0500 12/3/00, mlw wrote:

I honestly feel that it is wrong to take what others have shared and use
it for the basis of something you will not share, and I can't understand
how anyone could think differently.

Yeah, it really sucks when companies that are in buisness to make money by creating solutions and support for end users take the hard work of volenteers, commit resources to extending and enhancing that work, and make that work more accessable end users (for a fee).

Maybe it's unfair that the people at the bottom of that chain don't reap a percentage of the revenue generated at the top, but those people were free to read the license of the product they were contributing to.

Ironically, the GPL protects the future income a programmer much bettter than the BSD license, becuase under the GPL the original author can sell the code to a commercial enterprise who otherwise would not have been able to use it. Even more ironically, the GPL doesn't prevent 3rd parties from feeding at the trough as long as they DON'T extend and enhance the product. (Though Red Hat and friends donate work back to maintain community support.)

To me, Open Source is about admitting that the Computer Science field is in it's infancy, and the complex systems we're building today are the fundamental building blocks of tomorrow's systems. It is about exchanging control for adoption, a trade-off that has millions of case studies.

Think Different,
-pmb

--
"Every time you provide an option, you're asking the user to make a decision.
That means they will have to think about something and decide about it.
It's not necessarily a bad thing, but, in general, you should always try to
minimize the number of decisions that people have to make."
http://joel.editthispage.com/stories/storyReader$51

On Sun, 3 Dec 2000, Hannu Krosing wrote:

The Hermit Hacker wrote:

On Sat, 2 Dec 2000, Don Baccus wrote:

I *am* one of those volunteers

Yes, I well remember you screwing up PG 7.0 just before beta, without bothering
to test your code, and leaving on vacation.

You were irresponsible then, and you're being irresponsible now.

Okay, so let me get this one straight ... it was irresponsible for him to
put code in that was broken the last time, but it wouldn't be
irresponsible for us to release code that we don't feel is ready this
time? *raised eyebrow*

Just want to get this straight, as it kinda sounds hypocritical to me, but
want to make sure that I understand before I fully arrive at that
conclusion ... :)

IIRC, this thread woke up on someone complaining about PostgreSQl inc
promising
to release some code for replication in mid-october and asking for
confirmation
that this is just a schedule slip and that the project is still going on
and
going to be released as open source.

What seems to be the answer is: "NO, we will keep the replication code
proprietary".

I have not seen this answer myself, but i've got this impression from
the contents
of the whole discussion.

Do you know if this is the case ?

If this is the impression that someone gave, I am shocked ... Thomas
himself has already posted stating that it was a scheduale slip on his
part. Vadim did up the software days before the Oracle OpenWorld
conference, but it was a very rudimentary implementation. At the show,
Thomas dove in to build a basic interface to it, and, as time permits, has
been working on packaging to get it into contrib before v7.1 is released
...

I've been trying to follow this thread, and seem to have missed where
someone arrived at the conclusion that we were proprietarizing(word?) this
... we do apologize that it didn't get out mid-October, but it is/was
purely a scheduale slip ...

On Sun, 3 Dec 2000, mlw wrote:

On Sun, 3 Dec 2000, Gary MacDougall wrote:

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

Thats a given.

okay, then now I'm confused ... neither SePICK or erServer are derived
from uncompensated parties ... they work over top of PgSQL, but are not
integrated into them, nor have required any changes to PgSQL in order to
make it work ...

... so, where is this whole outcry coming from?

On Sun, 3 Dec 2000, Don Baccus wrote:

At 11:00 PM 12/2/00 -0800, Vadim Mikheev wrote:

There is risk here. It isn't so much in the fact that PostgreSQL, Inc
is doing a couple of modest closed-source things with the code. After
all, the PG community has long acknowleged that the BSD license would
allow others to co-op the code and commercialize it with no obligations.

It is rather sad to see PG, Inc. take the first step in this direction.

How long until the entire code base gets co-opted?

I totaly missed your point here. How closing source of ERserver is related
to closing code of PostgreSQL DB server? Let me clear things:

(not based on WAL)

That's wasn't clear from the blurb.

Still, this notion that PG, Inc will start producing closed-source products
poisons the well. It strengthens FUD arguments of the "open source can't
provide enterprise solutions" variety. "Look, even PostgreSQL, Inc realizes
that you must follow a close sourced model in order to provide tools for
the corporate world."

Don ... have you never worked for a client that has paid you to develop a
product for them? Have you taken the work you did for them, that they
paid for, and shoved it out into the community to use for free? Why would
we do anything any differently?

Your clients ask you to develop something for them as an extension to
PgSQL (its extensible, ya know?) that can be loaded as a simple module
(ala IPMeter) that gives them a competitive advantage over their
competitors, but that doesn't require any changes to the physical backend
to implement ... would you refuse their money? or would you do like
PgSQL, Inc is doing, where we do a risk-analysis of the changes and work
with the client to make use of the "competitive advantage" it gives them
for a period of time prior to releasing it open source?

Geoff explains it much better then I do, from a business perspective, but
any extension/application that PgSQL, Inc develops for our clients has a
life-span on it ... after which, keeping it in track with what is being
developed would cost more then the competitive advantage it gives the
clients ... sometimes, that is 0, some times, 6 months ... extreme cases,
24 months ...

Nobody is going to pay you to develop X if you are going to turn around
and give it for free to their competitor ... it makes no business. In
alot of cases, making these changes benefits the project as some of the
stuff that is required for them get integrated into the backend ...

On Sun, Dec 03, 2000 at 08:49:09PM -0400, The Hermit Hacker wrote:

On Sun, 3 Dec 2000, Hannu Krosing wrote:

IIRC, this thread woke up on someone complaining about PostgreSQl inc
promising
to release some code for replication in mid-october and asking for
confirmation
that this is just a schedule slip and that the project is still going on
and
going to be released as open source.

That would be me asking the question, as a reply to Don's concern regarding
the 'prorietary extension on a 24 mo. release delay'

What seems to be the answer is: "NO, we will keep the replication code
proprietary".

I have not seen this answer myself, but i've got this impression from
the contents
of the whole discussion.

Do you know if this is the case ?

If this is the impression that someone gave, I am shocked ... Thomas
himself has already posted stating that it was a scheduale slip on his
part.

Actually, Thomas said:

Thomas> Hmm. What has kept replication from happening in the past? It
Thomas> is a big job and difficult to do correctly. It is entirely my
Thomas> fault that you haven't seen the demo code released; I've been
Thomas> packaging it to make it a bit easier to work with.

I noted the use of the words "demo code" rather than "core code". That
bothered (and still bothers) me, but I didn't reply at the time,
since there was already enough heat in this thread. I'll take your
interpretation to mean it's just a matter of semantics.

[...] Vadim did up the software days before the Oracle OpenWorld
conference, but it was a very rudimentary implementation. At the show,
Thomas dove in to build a basic interface to it, and, as time permits, has
been working on packaging to get it into contrib before v7.1 is released
...

I've been trying to follow this thread, and seem to have missed where
someone arrived at the conclusion that we were proprietarizing(word?) this
... we do apologize that it didn't get out mid-October, but it is/was
purely a scheduale slip ...

Mixture of the silent schedule slip on the core code, and the explicit
statement on the erserver.com page regarding the 'proprietary extensions'
with a delayed source release.

The biggest problem I see with having core developers making proprietary
extensions is the potentional for conflict of interest when and if
some of us donate equivalent code to the core. The core developers who
have also done proprietary versions will have to be very cautious
when working on such code. They're in a bind, with two parts. First,
they have obligations to their employer and their employer's partners
to not release the closed work early. Second, possibly ignoring such
independent extensions, or even actively excluding them for the core,
in favor of their own code. The core developers _do_ have a bit of a
track record favoring each others code over external code, as is natural:
we all trust work more from sources we know better, especially when that
source is ourselves. But this favoratism could work against the earliest
possible open solution.

I'm still anxious to see the core patches needed to support replication.
Since you've leaked that they work going back to v6.5, I have a feeling
the approach may not be the one I was hoping for.

Ross

I'm still anxious to see the core patches needed to support replication.
Since you've leaked that they work going back to v6.5, I have a feeling
the approach may not be the one I was hoping for.

There are no core patches required to support replication. This has been
said already, but perhaps lost in the noise.

- Thomas

I'm agreeing with the people like SePICK and erServer.
I'm only being sort of cheeky in saying that they wouldn't have had a
product had
it not been for the Open Source that they are leveraging off of.
Making money? I don't know what they're plans are, but at some point I would
fully expect *someone* to make money.

----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: "Gary MacDougall" <gary@freeportweb.com>
Cc: "mlw" <markw@mohawksoft.com>; "Hannu Krosing" <hannu@tm.ee>; "Thomas
Lockhart" <lockhart@alumni.caltech.edu>; "Don Baccus"
<dhogaza@pacifier.com>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Sunday, December 03, 2000 7:53 PM
Subject: Re: [HACKERS] beta testing version

Correct me if I'm wrong but in the last 3 years what company that you
know of didn't consider an IPO part of the "business and such". Most
tech companies that have been formed in the last 4 - 5 years have one
thing on the brain--IPO. It's the #1 thing (sadly) that they care about.
I only wished these companies cared as much about *creating* and
inovation more than they cared about going public...

g.

On Sun, 3 Dec 2000, Ross J. Reedstrom wrote:

If this is the impression that someone gave, I am shocked ... Thomas
himself has already posted stating that it was a scheduale slip on his
part.

Actually, Thomas said:

Thomas> Hmm. What has kept replication from happening in the past? It
Thomas> is a big job and difficult to do correctly. It is entirely my
Thomas> fault that you haven't seen the demo code released; I've been
Thomas> packaging it to make it a bit easier to work with.

I noted the use of the words "demo code" rather than "core code". That
bothered (and still bothers) me, but I didn't reply at the time,
since there was already enough heat in this thread. I'll take your
interpretation to mean it's just a matter of semantics.

there is nothing that we are developing at this date that is *core* code
... the "demo code" that we are going to be putting into contrib is a
simplistic version, and the first cut, of what we are developing ... like
everything in contrib, it will be hack-on-able, extendable, etc ...

I'm still anxious to see the core patches needed to support
replication. Since you've leaked that they work going back to v6.5, I
have a feeling the approach may not be the one I was hoping for.

this is where the 'confusion' appears to be arising .. there are no
*patches* ... anything that will require patches to the core server will
almost have to be put to the open source or we hit problems where
development continues without us ... what we are doing with replication
requires *zero* patches to the server, it is purely a third-party
application ...

On Sun, 3 Dec 2000, Gary MacDougall wrote:

I'm agreeing with the people like SePICK and erServer.
I'm only being sort of cheeky in saying that they wouldn't have had a
product had
it not been for the Open Source that they are leveraging off of.

So, basically, if I hadn't pulled together Thomas, Bruce and Vadim 5 years
ago, when Jolly and Andrew finished their graduate thesis, and continued
to provide the resources required to bring PgSQL from v1.06 to now, we
wouldn't be able to use that as a basis for third party applications
... pretty much, ya, that sums it up ...

----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: "Gary MacDougall" <gary@freeportweb.com>
Cc: "mlw" <markw@mohawksoft.com>; "Hannu Krosing" <hannu@tm.ee>; "Thomas
Lockhart" <lockhart@alumni.caltech.edu>; "Don Baccus"
<dhogaza@pacifier.com>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Sunday, December 03, 2000 7:53 PM
Subject: Re: [HACKERS] beta testing version

On Sun, 3 Dec 2000, Gary MacDougall wrote:

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

Thats a given.

okay, then now I'm confused ... neither SePICK or erServer are derived
from uncompensated parties ... they work over top of PgSQL, but are not
integrated into them, nor have required any changes to PgSQL in order to
make it work ...

... so, where is this whole outcry coming from?

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

On Sun, Dec 03, 2000 at 08:53:08PM -0400, The Hermit Hacker wrote:

On Sun, 3 Dec 2000, Gary MacDougall wrote:

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

Thats a given.

okay, then now I'm confused ... neither SePICK or erServer are derived
from uncompensated parties ... they work over top of PgSQL, but are not
integrated into them, nor have required any changes to PgSQL in order to
make it work ...

... so, where is this whole outcry coming from?

This paragraph from erserver.com:

eRServer development is currently concentrating on core, universal
functions that will enable individuals and IT professionals
to implement PostgreSQL ORDBMS solutions for mission critical
datawarehousing, datamining, and eCommerce requirements. These
initial developments will be published under the PostgreSQL Open
Source license, and made available through our sites, Certified
Platinum Partners, and others in PostgreSQL community.

led me (and many others) to believe that this was going to be a tighly
integrated service, requiring code in the PostgreSQL core, since that's the
normal use of 'core' around here.

Now that I know it's a completely external implementation, I feel bad about
griping about deadlines. I _do_ wish I'd known this _design choice_ a bit
earlier, as it impacts how I'll try to do some things with pgsql, but that's
my own fault for over interpreting press releases and pre-announcements.

Ross

On Sun, 3 Dec 2000, Ross J. Reedstrom wrote:

On Sun, Dec 03, 2000 at 08:53:08PM -0400, The Hermit Hacker wrote:

On Sun, 3 Dec 2000, Gary MacDougall wrote:

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do what
ever you want.

Thats a given.

okay, then now I'm confused ... neither SePICK or erServer are derived
from uncompensated parties ... they work over top of PgSQL, but are not
integrated into them, nor have required any changes to PgSQL in order to
make it work ...

... so, where is this whole outcry coming from?

This paragraph from erserver.com:

eRServer development is currently concentrating on core, universal
functions that will enable individuals and IT professionals
to implement PostgreSQL ORDBMS solutions for mission critical
datawarehousing, datamining, and eCommerce requirements. These
initial developments will be published under the PostgreSQL Open
Source license, and made available through our sites, Certified
Platinum Partners, and others in PostgreSQL community.

led me (and many others) to believe that this was going to be a tighly
integrated service, requiring code in the PostgreSQL core, since that's the
normal use of 'core' around here.

Now that I know it's a completely external implementation, I feel bad about
griping about deadlines. I _do_ wish I'd known this _design choice_ a bit
earlier, as it impacts how I'll try to do some things with pgsql, but that's
my own fault for over interpreting press releases and pre-announcements.

Apologies from our side as well ... failings on the english language and
choice of said on our side ... the last thing that we want to do is have
to maintain patches across multiple versions for stuff that is core to the
server ... Thomas/Vadim can easily correct me if I've missed something,
but to the best of my knowledge, from our many discussions, anything that
is *core* to the PgSQL server itself will always be released similar to
any other project (namely, tested and open) ... including hooks for any
proprietary projects ... the sanctity of the *core* server is *always*
foremost in our minds, no matter what other projects we are working on ...

bingo.

Not just third-party app's, but think of all the vertical products that
include PG...
I'm right now wondering if TIVO uses it?

You have to think that PG will show up in some pretty interesting money
making products...

So yes, had you not got the ball rolling.... well, you know what I'm saying.

g.

----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: "Gary MacDougall" <gary@freeportweb.com>
Cc: "mlw" <markw@mohawksoft.com>; "Hannu Krosing" <hannu@tm.ee>; "Thomas
Lockhart" <lockhart@alumni.caltech.edu>; "Don Baccus"
<dhogaza@pacifier.com>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Sunday, December 03, 2000 10:18 PM
Subject: Re: [HACKERS] beta testing version

On Sun, 3 Dec 2000, Gary MacDougall wrote:

I'm agreeing with the people like SePICK and erServer.
I'm only being sort of cheeky in saying that they wouldn't have had a
product had
it not been for the Open Source that they are leveraging off of.

So, basically, if I hadn't pulled together Thomas, Bruce and Vadim 5 years
ago, when Jolly and Andrew finished their graduate thesis, and continued
to provide the resources required to bring PgSQL from v1.06 to now, we
wouldn't be able to use that as a basis for third party applications
... pretty much, ya, that sums it up ...

----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: "Gary MacDougall" <gary@freeportweb.com>
Cc: "mlw" <markw@mohawksoft.com>; "Hannu Krosing" <hannu@tm.ee>; "Thomas
Lockhart" <lockhart@alumni.caltech.edu>; "Don Baccus"
<dhogaza@pacifier.com>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Sunday, December 03, 2000 7:53 PM
Subject: Re: [HACKERS] beta testing version

On Sun, 3 Dec 2000, Gary MacDougall wrote:

If you write a program which stands on its own, takes no work from
uncompensated parties, then you have the unambiguous right to do

what

ever you want.

Thats a given.

okay, then now I'm confused ... neither SePICK or erServer are derived
from uncompensated parties ... they work over top of PgSQL, but are

not

integrated into them, nor have required any changes to PgSQL in order

to

make it work ...

... so, where is this whole outcry coming from?

Marc G. Fournier ICQ#7615664 IRC Nick:

Scrappy

Systems Administrator @ hub.org
primary: scrappy@hub.org secondary:

scrappy@{freebsd|postgresql}.org

At 09:42 PM 12/3/00 -0600, Ross J. Reedstrom wrote:

This paragraph from erserver.com:

eRServer development is currently concentrating on core, universal
functions that will enable individuals and IT professionals
to implement PostgreSQL ORDBMS solutions for mission critical
datawarehousing, datamining, and eCommerce requirements. These
initial developments will be published under the PostgreSQL Open
Source license, and made available through our sites, Certified
Platinum Partners, and others in PostgreSQL community.

led me (and many others) to believe that this was going to be a tighly
integrated service, requiring code in the PostgreSQL core, since that's the
normal use of 'core' around here.

Right. This is a big source of misunderstanding. There's still the fact
that 50% of the PG steering committee that are involved in [partially] closed
source development based on PG, though. This figure disturbs me.

50% is a lot. It's like ... half, right? Or did I miss something in the
conversion?

This represents significant change from the past where 0%, AFAIK, were
involved in closed source PG add-ons.

Now that I know it's a completely external implementation, I feel bad about
griping about deadlines. I _do_ wish I'd known this _design choice_ a bit
earlier, as it impacts how I'll try to do some things with pgsql, but that's
my own fault for over interpreting press releases and pre-announcements.

IF 50% of the steering committee is to embark on such a task in a closed source
or semi-closed source development model, it would seem common courtesy to inform the
community of the facts as early as they were decided upon.

In fact, it might seem to be common courtesy to float the notion in the community,
to gauge reaction and to build support, before finalizing such a decision.

AFAIC this arrived out of no where, a sort of stealth "50% of the steering committee
has decided to embark on a semi-proprietary solution to the replication problem that
you won't see as open source for [up to] two years after its completion".

That's a paradigm shift. Whether right or wrong, there's a responsibility to
communicate the fact that 50% of the steering committee has decided to partially
abandon the open source development model for one that is (in some cases) closed
for two years and (in other cases) forever.

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

At 11:59 PM 12/3/00 -0400, The Hermit Hacker wrote:

the sanctity of the *core* server is *always*
foremost in our minds, no matter what other projects we are working on ...

What happens if financially things aren't entirely rosy with your company?
The problem in taking itty-bitty steps in this direction is that you're
involving outside money interests that don't necessarily adhere to this
view.

Having taken the first steps to a proprietary, closed source future, would
you pledge to bankrupt your company rather than accept a large captital
investment with an ROI based on proprietary extensions to the core that
might not be likely to come out of the non-tainted side of the development
house?

Would your company sign a contract to that effect with independent parties,
i.e. that it would never violate the sanctity of the *core*? Even if it means
you go broke? And that your investors go broke?

Or would your investors prefer you not make such a formal committment, in order
to keep options open if things don't go well?

(in the early 80's my company received a total of $8,000,000 in pre-IPO
capital investments, so I have some experience with the expectations of investors.
It tends to make me a bit paranoid. I'm not the only COO to have such experiences
while living the life).

What happens in two years if those investors in eRServer haven't gotten adequate
return on their investment? Do you have a formal agreement that the source will
be released regardless? Can the community inspect the agreement so we can judge
for ourselves whether or not this assurance is adequately backed by contract
language?

Are your agreements Open Source? :)

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.