Postgresql and multithreading

Anuradha Ratnaweera wrote:

Is there any plans to make postgresql multithreading?

Thanks in advance (and also for all who commented to my question
regarding replication.)

Anuradha

NB: please don't open fire to declare war on whether multithreading is
needed for PGSql or not. I am just expecting a black and white answer
from the `authorities' ;)

We don't think it is needed, except perhaps for Win32 and Solaris, which
have slow process creation times.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

Is there any plans to make postgresql multithreading?

We don't think it is needed, except perhaps for Win32 and Solaris, which
have slow process creation times.

Thanks, Bruce. But what I want to know is whether multithreading is
likely to get into in postgresql, say somewhere in 8.x, or even in 9.x?
(as they did with Apache). Are there any plans to do so, or is postgres
going to remain rather a multi-process application?

Anuradha

--

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

One nice thing about egotists: they don't talk about other people.

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

Is there any plans to make postgresql multithreading?

Thanks in advance (and also for all who commented to my question
regarding replication.)

Anuradha

NB: please don't open fire to declare war on whether multithreading is
needed for PGSql or not. I am just expecting a black and white answer
from the `authorities' ;)

This has been discussed at length in the past. The answer has always been
no. Consult the archives for extensive heated discussion :-).

Gavin

Anuradha Ratnaweera wrote:

On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

Is there any plans to make postgresql multithreading?

We don't think it is needed, except perhaps for Win32 and Solaris, which
have slow process creation times.

Thanks, Bruce. But what I want to know is whether multithreading is
likely to get into in postgresql, say somewhere in 8.x, or even in 9.x?
(as they did with Apache). Are there any plans to do so, or is postgres
going to remain rather a multi-process application?

It may be optional some day, most likely for Win32 at first, but we see
little value to it on most other platforms; of course, we may be wrong.
I am also not sure if it is a big win on Apache either; I think the
jury is still out on that one, hence the slow adoption of 2.X, and we
don't want to add threads and make a mess of the code or slow it down,
which does often happen.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

... what I want to know is whether multithreading is likely to get
into in postgresql, say somewhere in 8.x, or even in 9.x?

It may be optional some day, most likely for Win32 at first, but we see
little value to it on most other platforms; of course, we may be wrong.

In that case, I wonder if it is worth folking a new project to add
threading support to the backend? Of course, keeping in sync with the
original would be lot of work.

In that way, one should be able to test the hypothesis (whether threads
improve things, or the other way round - if one likes it it that way :))
without messing around with stable postgres code, as they did and do
with postgresql-R.

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

I am also not sure if it is a big win on Apache either; I think the
jury is still out on that one, hence the slow adoption of 2.X,

As far as we are concened, it is the stability, rather than speed which
still keeps us in 1.3.

and we don't want to add threads and make a mess of the code or slow
it down, which does often happen.

Fully agreed.

Anuradha

--

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

Equality is not when a female Einstein gets promoted to assistant
professor; equality is when a female schlemiel moves ahead as fast as a
male schlemiel.
-- Ewald Nyquist

Anuradha Ratnaweera wrote:

On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

... what I want to know is whether multithreading is likely to get
into in postgresql, say somewhere in 8.x, or even in 9.x?

It may be optional some day, most likely for Win32 at first, but we see
little value to it on most other platforms; of course, we may be wrong.

In that case, I wonder if it is worth folking a new project to add
threading support to the backend? Of course, keeping in sync with the
original would be lot of work.

Probably not, but you can try.

In that way, one should be able to test the hypothesis (whether threads
improve things, or the other way round - if one likes it it that way :))
without messing around with stable postgres code, as they did and do
with postgresql-R.

I guess.

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

We don't think we change the license, and we are happy with BSD. It
certainly will never be merged in with a GPL, I can say that for sure.

I am also not sure if it is a big win on Apache either; I think the
jury is still out on that one, hence the slow adoption of 2.X,

As far as we are concened, it is the stability, rather than speed which
still keeps us in 1.3.

You could easily lose stability with threads -- don't think they are a
free ride --- they aren't, and no, I don't feel like regurgitating what
is already a 'thread' link on the TODO list.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

... what I want to know is whether multithreading is likely to get
into in postgresql, say somewhere in 8.x, or even in 9.x?

It may be optional some day, most likely for Win32 at first, but we see
little value to it on most other platforms; of course, we may be wrong.

In that case, I wonder if it is worth folking a new project to add
threading support to the backend? Of course, keeping in sync with the
original would be lot of work.

http://sourceforge.net/projects/mtpgsql

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

Do you mean 'relicence the forked copy'?

Gavin

On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote:

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

Do you mean 'relicence the forked copy'?

Nope. To keep the `original' code licence as it is and to release the
changes GPL? Is the question sane at first place?

Anuradha

--

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

You got to be very careful if you don't know where you're going,
because you might not get there.
-- Yogi Berra

Anuradha Ratnaweera wrote:

On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote:

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

Do you mean 'relicence the forked copy'?

Nope. To keep the `original' code licence as it is and to release the
changes GPL? Is the question sane at first place?

That would be a pretty big mess, I think. People would add your patch
to our BSD code and it would be GPL. It could be done, of course.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Let me add one more thing on this "thread". This is one email in a long
list of "Oh, gee, you aren't using that wizz-bang new
sync/thread/aio/raid/raw feature" discussion where someone shows up and
wants to know why. Does anyone know how to address these, efficiently?

If we discuss it, it ends up causing a lot of effort on our part for the
requestor to finally say, "Oh, gee, I didn't realize that." It isn't
our job to explain that wizz-bang isn't always great. Maybe that should
be our reply, "Wizz-bang isn't always great" and leave it at that. Of
course, some will leave thinking we are just idiots, but then again, it
takes one to know one. It is sort of like walking into a chess match
and asking Bobby Fisher why he didn't move that pawn. :-)

---------------------------------------------------------------------------

Anuradha Ratnaweera wrote:

On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote:

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

Do you mean 'relicence the forked copy'?

Nope. To keep the `original' code licence as it is and to release the
changes GPL? Is the question sane at first place?

Anuradha

--

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

You got to be very careful if you don't know where you're going,
because you might not get there.
-- Yogi Berra

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote:

Let me add one more thing on this "thread". This is one email in a
long list of "Oh, gee, you aren't using that wizz-bang new
sync/thread/aio/raid/raw feature" discussion where someone shows up
and wants to know why. Does anyone know how to address these,
efficiently?

If somebody pops up asks such dumb questions without even looking at the
FAQ, it is bad, if not idiotic, because it takes useful time away from
the developers.

But my question was not `why don't you implement this feature?`, but `do
you have plans to implement this feature in the future?', and in the
open source spirit of `if something is not there, go implement it
yourself - without troubling developers' ;)

Also, I have read the section 1.9 of the developers FAQ (Why don't we
use threads in the backend?) long, long ago.

If we discuss it, it ends up causing a lot of effort on our part for
the requestor to finally say, "Oh, gee, I didn't realize that."

Please don't. See the "NB" at end of my first mail of this thread.

Anuradha

--

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

QOTD:
"I'll listen to reason when it comes out on CD."

On 16 Oct 2002 at 1:25, Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

Thanks, Bruce. But what I want to know is whether multithreading is
likely to get into in postgresql, say somewhere in 8.x, or even in 9.x?
(as they did with Apache). Are there any plans to do so, or is postgres
going to remain rather a multi-process application?

It may be optional some day, most likely for Win32 at first, but we see
little value to it on most other platforms; of course, we may be wrong.
I am also not sure if it is a big win on Apache either; I think the

Well, I have done some stress testing on 1.3.26 and 2.0.39. Under same hardware
and network setup and same test case, 1.3.26 maxed at 475-500 requests/sec and
2.0.39 gave flat 800 requests/sec.

Yes, under light load, there is hardly any difference. But Apache2 series is
definitely an improvement.

jury is still out on that one, hence the slow adoption of 2.X, and we
don't want to add threads and make a mess of the code or slow it down,
which does often happen.

Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us'
syndrome, as far as I can see from news. Once linux distros start shipping with
apache 2.x series *only*, the upgrade cycle will start rolling, I guess.

Bye
Shridhar

--
Programming Department: Mistakes made while you wait.

On 16 Oct 2002 at 15:40, Gavin Sherry wrote:

In that case, I wonder if it is worth folking a new project to add
threading support to the backend? Of course, keeping in sync with the
original would be lot of work.

http://sourceforge.net/projects/mtpgsql

Last discussion that happened for multithreading was not to add threads per
connection like mysql, but to split tasks between threads so that IO blocking
and efficiently using SMP abilities would be taken care off IIRC..

One thread per connection isn't going to change much, at least for mainstream
postgresql. For embedded, it might be a necessity..

Bye
Shridhar

--
QOTD: "You're so dumb you don't even have wisdom teeth."

Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

<snip>

Nope. To keep the `original' code licence as it is and to release the
changes GPL? Is the question sane at first place?

That would be a pretty big mess, I think. People would add your patch
to our BSD code and it would be GPL. It could be done, of course.

Don't think so. The patches would be "derived code" that only exist
because of the BSD licensed PostgreSQL base.

Being "derived code" they'd have to be released as BSD and GPL wouldn't
enter the picture, regardless if they're released separately as add-on
patches or not.

:-)

Regards and best wishes,

Justin Clift

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

On Wed, 2002-10-16 at 01:27, Shridhar Daithankar wrote:

Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us'
syndrome, as far as I can see from news. Once linux distros start shipping with
apache 2.x series *only*, the upgrade cycle will start rolling, I guess.

I think that's part of it. I think the other part is that by the time
you're getting to huge r/s numbers, typical web site bandwidth is
already used up. So, what's the point in adding more breathing room
when you don't have the bandwidth to use it anyways.

Greg

On Wed, 2002-10-16 at 04:34, Justin Clift wrote:

Bruce Momjian wrote:

Anuradha Ratnaweera wrote:

<snip>

Nope. To keep the `original' code licence as it is and to release the
changes GPL? Is the question sane at first place?

That would be a pretty big mess, I think. People would add your patch
to our BSD code and it would be GPL. It could be done, of course.

Don't think so. The patches would be "derived code" that only exist
because of the BSD licensed PostgreSQL base.

Being "derived code" they'd have to be released as BSD and GPL wouldn't
enter the picture, regardless if they're released separately as add-on
patches or not.

I'm pretty sure BSD allows you to relicense derived code as you see fit.
However, any derived project that was released GPL would have a hell of
a time ever getting put back into the main source (short of
relicensing).

Robert Treat

On Wed, 2002-10-16 at 16:37, Robert Treat wrote:

I'm pretty sure BSD allows you to relicense derived code as you see fit.
However, any derived project that was released GPL would have a hell of
a time ever getting put back into the main source (short of
relicensing).

Exactly. This is one of the reasons why BSD license is very much liked
by proprietary software vendors (think MSFT), unlike the GPL which
doesn't allow someone to change and redistribute their work with
restrictive licenses.

Cheers,
Tycho

(BTW: I'm not asking to change the license of Postgresql, I know the -
dogmatic - answer to that one. So please don't misunderstand my mail)

--
Tycho Fruru tycho@fruru.com
"Prediction is extremely difficult. Especially about the future."
- Niels Bohr

Bruce Momjian wrote:

Let me add one more thing on this "thread". This is one email in a long
list of "Oh, gee, you aren't using that wizz-bang new
sync/thread/aio/raid/raw feature" discussion where someone shows up and
wants to know why. Does anyone know how to address these, efficiently?

You've brought up good points here. I'm sure that you consider me guilty of
this with regard to my aio discussions. So I might offer a few specific
suggestions.

1) Someone's taking the time to generate a summary of the current thinking
with regard to a particular "whiz-bang" feature. - I can do this as a first
pass for aio, if you think it's a good idea.

2) Including the pros and cons of the feature/implementation and how close
the group is to deciding whether something would be worth doing. - I can
also do this.

3) A set of criteria that would need to be met or demonstrated before a
feature would be considered good enough for inclusion into the main code.

If there was a separate section of the document "Developer's FAQ" that
handled this, this would help.

4) A development/feature philosophy section. Maybe three or four paragraphs
with one specific example: perhaps multi-threading.

5) I'd also suggest changing some specfics in the FAQ Section 1.2 where
there is currently:

* The usual process for source additions is:
*
* Review the TODO list.
* Discuss hackers the desirability of the fix/feature.
* How should it behave in complex circumstances?

Add here that a check should be made to the new section in the FAQ
described above. Also, section 1.1 has:

* Discussion on the patch typically happens here. If the patch adds a
* major feature, it would be a good idea to talk about it first on
* the HACKERS list, in order to increase the chances of it being
* accepted, as well as toavoid duplication of effort.

We should perhaps add here a section describing the phenomenon you describe
regarding "whiz-bang" features.

I tried to prepare as best I could before bringing anything forward to
HACKERS. In particular, I read the last two years of archives with anything
to do with the WAL log and looked at the current code, read the TODOs, read
a fair amount of discussions about aio. etc. So I was attempting to comply
with my interpretation of the process. Yet, despite these efforts, you no
doubt consider me guilty of creating unnecessary work, an outcome neither
of us desired.

I'm undeterred in my desire to come up with something meaningful and am
working on some interesting tests. I do, however, now know that the level
of scepticism and the standards of architectural purity are high. I
consider this good all around.

- Curtis

On Wed, 2002-10-16 at 23:08, Curtis Faith wrote:

Bruce Momjian wrote:
I tried to prepare as best I could before bringing anything forward to
HACKERS. In particular, I read the last two years of archives with anything
to do with the WAL log and looked at the current code, read the TODOs, read
a fair amount of discussions about aio. etc. So I was attempting to comply
with my interpretation of the process. Yet, despite these efforts, you no
doubt consider me guilty of creating unnecessary work, an outcome neither
of us desired.

But that "unneccessary work" resulted in Tom finding and fixing an
unintended behaviour (aka a performance bug) that prevented postgres
from ever doing more than 1 commit per disk revolution on non-lieing
SCSI disks ;)

I'm undeterred in my desire to come up with something meaningful and am
working on some interesting tests. I do, however, now know that the level
of scepticism and the standards of architectural purity are high. I
consider this good all around.

I still have big expectations for use of aio, especially considering
that at least for free OSes one is not forced to stop at the DB/OS
boundary, but are free to go and improve the os side implementation as
well if it is needed.

But still some empirical tests are probably needed - if we can keep IO
occupied for 99% in a meaningful way withou aio, then the time is
probably better spent on something else ;)

--------------------
Hannu