What's planned for 7.5?

Native Win32 is planned for it (whether it makes it or not is another
question, but it is the goal) ...

On Mon, 12 Jan 2004, ow wrote:

Hi,

Is this all that's planned for 7.5? (based on current TODO list)

-Change factorial to return a numeric (Gavin)
-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)
-Have psql \dn show only visible temp schemas using current_schemas()
-Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
-Allow psql \du to show groups, and add \dg for groups
-Allow pg_dump to dump CREATE CONVERSION (Christopher)
-Use dependency information to dump data in proper order
-Use background process to write dirty shared buffers to disk

Thanks

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

Mensaje citado por "Marc G. Fournier" <scrappy@postgresql.org>:

Native Win32 is planned for it (whether it makes it or not is another
question, but it is the goal) ...

Replication wasn't another BIG one?

--
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
---------------------------------------------------------
Mart�n Marqu�s | Programador, DBA
Centro de Telem�tica | Administrador
Universidad Nacional
del Litoral
---------------------------------------------------------

On Mon, 12 Jan 2004, Martin Marques wrote:

Mensaje citado por "Marc G. Fournier" <scrappy@postgresql.org>:

Native Win32 is planned for it (whether it makes it or not is another
question, but it is the goal) ...

Replication wasn't another BIG one?

Actually, I think it was PITR (Point in Time Recovery). Of course, since
that works by replaying log files into the database, it's a pretty good
way of setting up replication, which would likely follow a release or so
after PITR was implemented.

OTOH, there are some replication solutions already available, just no as
part of the core.

Mensaje citado por ow <oneway_111@yahoo.com>:

Hi,

Is this all that's planned for 7.5? (based on current TODO list)

-Change factorial to return a numeric (Gavin)
-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)
-Have psql \dn show only visible temp schemas using current_schemas()
-Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
-Allow psql \du to show groups, and add \dg for groups
-Allow pg_dump to dump CREATE CONVERSION (Christopher)
-Use dependency information to dump data in proper order
-Use background process to write dirty shared buffers to disk

For what I have just seen in Robert Treats "PostgreSQL Weekly News" most of
these issues are already solved in the CVS, and I guess that in a bit less than
a year that's left for the release of 7.5 many new things will appear.

Anyway, from http://developer.postgresql.org/todo.php I see in the URGENT part
this:

# Add replication of distributed databases [replication]

* Automatic failover
* Load balancing
* Master/slave replication
* Multi-master replication
* Partition data across servers
* Queries across databases or servers (two-phase commit)
* Allow replication over unreliable or non-persistent links
* http://gborg.postgresql.org/project/pgreplication/projdisplay.php

# Point-in-time data recovery using backup and write-ahead log
# Create native Win32 port,
* http://momjian.postgresql.org/main/writings/pgsql/win32.html

The first isn't new, but as for what I know, it's a new aproach that Jan has for
adding replication to the server.

The other 2 didn't get in the 7.4 release.

--
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
---------------------------------------------------------
Mart�n Marqu�s | Programador, DBA
Centro de Telem�tica | Administrador
Universidad Nacional
del Litoral
---------------------------------------------------------

On Mon, 12 Jan 2004, Martin Marques wrote:

Mensaje citado por ow <oneway_111@yahoo.com>:

Hi,

Is this all that's planned for 7.5? (based on current TODO list)

-Change factorial to return a numeric (Gavin)
-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)
-Have psql \dn show only visible temp schemas using current_schemas()
-Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
-Allow psql \du to show groups, and add \dg for groups
-Allow pg_dump to dump CREATE CONVERSION (Christopher)
-Use dependency information to dump data in proper order
-Use background process to write dirty shared buffers to disk

For what I have just seen in Robert Treats "PostgreSQL Weekly News" most of
these issues are already solved in the CVS, and I guess that in a bit less than
a year that's left for the release of 7.5 many new things will appear.

Anyway, from http://developer.postgresql.org/todo.php I see in the URGENT part
this:

# Add replication of distributed databases [replication]

* Automatic failover
* Load balancing
* Master/slave replication
* Multi-master replication
* Partition data across servers
* Queries across databases or servers (two-phase commit)
* Allow replication over unreliable or non-persistent links
* http://gborg.postgresql.org/project/pgreplication/projdisplay.php

# Point-in-time data recovery using backup and write-ahead log
# Create native Win32 port,
* http://momjian.postgresql.org/main/writings/pgsql/win32.html

The first isn't new, but as for what I know, it's a new aproach that Jan has for
adding replication to the server.

Nope, pgreplication has nothing to do with Jan ... he's working on
Slonik-1 ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

martin@bugs.unl.edu.ar (Martin Marques) writes:

Mensaje citado por "Marc G. Fournier" <scrappy@postgresql.org>:

Native Win32 is planned for it (whether it makes it or not is another
question, but it is the goal) ...

Replication wasn't another BIG one?

Well, there are two "replication things" going on:

1. The eRserv Java server software is already available on gBorg.

It isn't likely to go into the software distribution any time
soon because of its dependence on Java. It really is an extra
add-on.

2. Jan Wieck's work on SLONY-1

Many are keen to see the code, but it's not out yet. And it is
not self-evident that it will necessarily be ready to release in
conjunction with 7.5 (Not to say it _can't_ happen, but just that
we'll see the code when we see it...)

It is planned to be implemented in C, so it would presumably be
more suitable for inclusion in the main code than eRserv. But it
stands as "hope," and not certainty.

And none of this is ...

3. PITR (Point In Time Recovery).

Where a characteristic "hope" would be to archive WAL files and
use them to bring a failed DB up to date.

Jan has plans to do something similar but _much_ less
obscure/opaque with SLONY-1. We'll see what happens...

Another fairly important item not mentioned: Nested Transactions.
--
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/&gt;
Christopher Browne
(416) 646 3304 x124 (land)

ow <oneway_111@yahoo.com> writes:

Is this all that's planned for 7.5? (based on current TODO list)

If you think the TODO list has *anything* to do with release planning,
you fundamentally misunderstand the way things are done around here.

The TODO list is a list of things that are widely agreed to be problems
(though sometimes it only means "Bruce thinks this is a problem") and
therefore will probably get worked on at some point in the future.

What actually gets done for 7.5 will depend on which itches bother which
developers enough to get scratched in the next few months. We do not
have any central planning.

It is a safe bet that 7.5 will include some things mentioned in the
present TODO, as well as many things not mentioned in it; and that
many items mentioned in the present TODO will still remain undone.

regards, tom lane

Any chance we'll see the VACUUM delay patch (throttle) get into 7.5? I only
had the chance to try the first patch by Tom Lane and it was very good
already. I was hoping it gets into 7.4.1 but it didn't. :-(

I really need the VACUUM delay patch because my servers are begging to die
every time VACUUM runs. Please let it be in 7.5.

Thanks.

Stephen

"Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
news:5658.1073947557@sss.pgh.pa.us...

"Stephen" <jleelim@xxxxxxx.com> writes:

Any chance we'll see the VACUUM delay patch (throttle) get into 7.5? I only
had the chance to try the first patch by Tom Lane and it was very good
already. I was hoping it gets into 7.4.1 but it didn't. :-(

I really need the VACUUM delay patch because my servers are begging to die
every time VACUUM runs. Please let it be in 7.5.

The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
version, working.

The delay patch is simple enough to add in if you need it :-).
--
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/&gt;
Christopher Browne
(416) 646 3304 x124 (land)

Christopher Browne <cbbrowne@libertyrms.info> writes:

"Stephen" <jleelim@xxxxxxx.com> writes:

Any chance we'll see the VACUUM delay patch (throttle) get into 7.5?

The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
version, working.

Actually, I'm not sure that ARC should be considered to supersede the
usefulness of a per-page delay in VACUUM. ARC should prevent VACUUM
from trashing the contents of Postgres' shared buffer arena, but it
won't do much of anything to prevent VACUUM from trashing the kernel
buffer contents. And it definitely won't do anything to help if the
real problem is that you're short of disk bandwidth and VACUUM's extra
I/O demand pushes your total load over the knee of the response-time
curve. What you need then is a throttle.

The original patch I posted was incomplete for a number of reasons,
but I think it may still be worth working on. Jan, any comments?

regards, tom lane

Tom Lane wrote:

Christopher Browne <cbbrowne@libertyrms.info> writes:

"Stephen" <jleelim@xxxxxxx.com> writes:

Any chance we'll see the VACUUM delay patch (throttle) get into 7.5?

The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
version, working.

Actually, I'm not sure that ARC should be considered to supersede the
usefulness of a per-page delay in VACUUM. ARC should prevent VACUUM
from trashing the contents of Postgres' shared buffer arena, but it
won't do much of anything to prevent VACUUM from trashing the kernel
buffer contents. And it definitely won't do anything to help if the
real problem is that you're short of disk bandwidth and VACUUM's extra
I/O demand pushes your total load over the knee of the response-time
curve. What you need then is a throttle.

The original patch I posted was incomplete for a number of reasons,
but I think it may still be worth working on. Jan, any comments?

I agree that there is considerable value in IO distribution. As such I
already have the basics of the Background Writer in there.

I left out the vacuum delay since I thought it was good enough to proove
that there is low hanging fruit, but that it was far from what I'd call
a solution. Ideally Vacuum would coordinate it's IO not only against
some GUC variables, but also against the BGWriter+BufStrategy combo.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

The vacuum delay patch is not the ideal solution but it worked like a charm
on my servers. I really need the vacuum delay patch or a better solution in
7.5. I'm getting millions of requests a month and running VACUUM without the
patch makes PostgreSQL useless for many consecutive hours. Not quite the
24/7 system I was hopping for. :-(

Unfortunately, it's rather difficult to patch so many machines as my entire
system runs on Redhat RPMs. I'm really hopping to see a solution to this
VACUUM problem in 7.5. I've been waiting for this fix for over 3 years and
now it's almost there.

Will this problem get addressed in the not so official TODO list?

Thanks and keep up the good work!

Stephen

"Jan Wieck" <JanWieck@Yahoo.com> wrote in message
news:400544E3.8030709@Yahoo.com...

Christopher Browne wrote:

2. Jan Wieck's work on SLONY-1

Many are keen to see the code, but it's not out yet. And it is
not self-evident that it will necessarily be ready to release in
conjunction with 7.5 (Not to say it _can't_ happen, but just that
we'll see the code when we see it...)

It is planned to be implemented in C, so it would presumably be
more suitable for inclusion in the main code than eRserv. But it
stands as "hope," and not certainty.

I've just made one major step backward on that. Discovered that my first
thread model was deadlock prone, so I better throw that away instead of
building a lot of code on top of it.

What I currently do is documenting the Slony-I ERD and the new thread
model I have in mind. This will be a document in work, but I plan to
have something readable by the end of this week, latest mid next week. I
will create a mailing list for Slony-I on gborg so we can start
discussing the implementation details.

About the inclusion of a replication solution into the core distributon.
The much I personally would be proud to see this one added, as a CORE
member I do not see any of the replication "things" fit. All of them,
and neither Slony-I nor Slony-II will be any different here, have pros
and cons, none is the "one size that fits all" magic solution. To select
one of the replication projects that high above all the others that it
will be added to the core distribution, it should be really outstanding
and general purpose. I think that Slony in the end will be outstanding,
but only for what it was designed for.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Stephen wrote:

The vacuum delay patch is not the ideal solution but it worked like a charm
on my servers. I really need the vacuum delay patch or a better solution in
7.5. I'm getting millions of requests a month and running VACUUM without the
patch makes PostgreSQL useless for many consecutive hours. Not quite the
24/7 system I was hopping for. :-(

Unfortunately, it's rather difficult to patch so many machines as my entire
system runs on Redhat RPMs. I'm really hopping to see a solution to this
VACUUM problem in 7.5. I've been waiting for this fix for over 3 years and
now it's almost there.

Will this problem get addressed in the not so official TODO list?

Well, I had a few different "versions" of vacuum delay stuff out as
patches, together with ARC and the beginnings of the background writer.
Instead of getting some numbers on those, the whole discussion got stuck
in differences about how we actually let the background writer tell the
kernel "do something" ... the whole sync(), fsync(), fdatasync(),
fadvise() discussion.

I don't have the time to make enough different attempts to find the one
that pleases all. My argument still is that all this IO throttling and
IO optimizing is mainly needed for dedicated servers, because I think
that if you still run multiple services on one box you're not really in
trouble yet. So in the first round a configurable sync() approach would
do. So far nobody even agreed to that.

I currently have better to do. We do not have a big IO problem, we have
other problems, and I spend my time on solving them. If someone wants to
pick up the IO throttle problem, I am allways here to help, but I will
not waste my time with making patches nobody even gives a try.

Thanks and keep up the good work!

Sorry for the venting, but I needed that out.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

On Mon, Jan 19, 2004 at 08:12:28AM -0500, Jan Wieck wrote:

and cons, none is the "one size that fits all" magic solution. To select

Does anyone realy believe that there can be a one size fits all
solution? Heck, even Oracle and IBM offer a couple of different
systems, depending on what you need. (That also suggests that any
replication system need not always be shipped with the "basic"
distribution, but could instead be integrated into a larger,
postgresql_plus_enterprise_features.tgz or something like that.)

A

--
Andrew Sullivan | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to. That actually seems sort of quaint now.
--J.D. Baldwin

People,

I don't have the time to make enough different attempts to find the one
that pleases all. My argument still is that all this IO throttling and
IO optimizing is mainly needed for dedicated servers, because I think
that if you still run multiple services on one box you're not really in
trouble yet. So in the first round a configurable sync() approach would
do. So far nobody even agreed to that.

I won't claim expertise on the different sync algorithms. However, I do need
to speak up in support of Jan's assertion; the machines most likely to suffer
I/O choke are, or should be, dedicated machines. If someone's running 6
major server applications on a server with a 25GB database and a single
RAID-5 array, then they've got to expect some serious performance issues.

We currently have a lot of users running large databases on devoted servers,
though, and they can't vaccuum their databases during working hours because
the vacuum ties up the I/O for 10 minutes or more. It's a bad situation and
makes us look very bad in comparison to the proprietary databases, which have
largely solved this problem. Maybe the sync() approach isn't perfect, but
it's certainly better than not doing anything, particularly if it can be
turned off at startup time.

--
-Josh Berkus
Aglio Database Solutions
San Francisco

On Mon, 2004-01-19 at 08:37, Jan Wieck wrote:

but I will not waste my time with making patches nobody even gives a try.

I downloaded and tested your patches. I just didn't get results get
results that were put together well enough to present to the group. I
hope this doesn't fall by the wayside, it is IMHO, on of the critical
problems that needs to be solved.

Sorry for the venting, but I needed that out.

I understand. I'm sorry there wasn't more feedback as a result of your
work.

Andrew Sullivan wrote:

On Mon, Jan 19, 2004 at 08:12:28AM -0500, Jan Wieck wrote:

and cons, none is the "one size that fits all" magic solution. To select

Does anyone realy believe that there can be a one size fits all
solution? Heck, even Oracle and IBM offer a couple of different
systems, depending on what you need. (That also suggests that any
replication system need not always be shipped with the "basic"
distribution, but could instead be integrated into a larger,
postgresql_plus_enterprise_features.tgz or something like that.)

I don't, but consider Linux which can be configured to run on devices as
small as a wristwatch and as large as the the big irons...

--
dave

Josh Berkus wrote:

People,

I don't have the time to make enough different attempts to find the one
that pleases all. My argument still is that all this IO throttling and
IO optimizing is mainly needed for dedicated servers, because I think
that if you still run multiple services on one box you're not really in
trouble yet. So in the first round a configurable sync() approach would
do. So far nobody even agreed to that.

I won't claim expertise on the different sync algorithms. However, I do need
to speak up in support of Jan's assertion; the machines most likely to suffer
I/O choke are, or should be, dedicated machines. If someone's running 6
major server applications on a server with a 25GB database and a single
RAID-5 array, then they've got to expect some serious performance issues.

We currently have a lot of users running large databases on devoted servers,
though, and they can't vaccuum their databases during working hours because
the vacuum ties up the I/O for 10 minutes or more. It's a bad situation and
makes us look very bad in comparison to the proprietary databases, which have
largely solved this problem. Maybe the sync() approach isn't perfect, but
it's certainly better than not doing anything, particularly if it can be
turned off at startup time.

Thanks for the support Josh,

though the sync() issues of the background writer and vacuum might not
seem directly related, it all must be done in the same IO bandwidth.

So if we are to do this now, this would be my proposal:

* GUC parameter vacuum_cost_page_hit=1 is the cost for a page found
by vacuum on a buffer cache hit.

* GUC parameter vacuum_cost_page_miss=10 is the cost for a page
faulted in on behalf of vacuum.

* GUC parameter vacuum_cost_page_dirtied=20 is the cost for vacuum
marking a formerly clean page dirty.

* GUC parameter vacuum_cost_limit=200 is the amount of cost vacuum
can produce before napping.

* GUC parameter vacuum_cost_naptime=0 (by default the entire mechanism
disabled) is the number of milliseconds to nap when the cost limit
is reached.

* Pages faulted in on behalf of vacuum are placed onto the replacement
cache head for immediate eviction.

In addition to this, vacuum will yield while the background writer is
doing any work. The GUC option bgwriter_sync_method=none (or sync) will
control if the background writer will cause a smgr_sync() at the end of
a run.

Both, vacuum and the background writer, will yield to a checkpoint. With
a properly configured background writer, checkpoints do not cause major
responsetime spikes any more.

Anything forgotten? Tom?

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

ow wrote:

Hi,

Is this all that's planned for 7.5? (based on current TODO list)

-Change factorial to return a numeric (Gavin)
-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)
-Have psql \dn show only visible temp schemas using current_schemas()
-Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
-Allow psql \du to show groups, and add \dg for groups
-Allow pg_dump to dump CREATE CONVERSION (Christopher)
-Use dependency information to dump data in proper order
-Use background process to write dirty shared buffers to disk

Those dashes mean the items are done and are already in CVS. As to what
additional items we will have in 7.5, no one knows.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tom Lane wrote:

ow <oneway_111@yahoo.com> writes:

Is this all that's planned for 7.5? (based on current TODO list)

If you think the TODO list has *anything* to do with release planning,
you fundamentally misunderstand the way things are done around here.

The TODO list is a list of things that are widely agreed to be problems
(though sometimes it only means "Bruce thinks this is a problem") and
therefore will probably get worked on at some point in the future.

What actually gets done for 7.5 will depend on which itches bother which
developers enough to get scratched in the next few months. We do not
have any central planning.

It is a safe bet that 7.5 will include some things mentioned in the
present TODO, as well as many things not mentioned in it; and that
many items mentioned in the present TODO will still remain undone.

Yes, the TODO is just a collection of items that could be done in 7.5.
Some will be, and some new items not on the TODO list will be in 7.5.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)

Hey Bruce,

You probably should add 'Dump LOB comments in custom dump format' to the
todo. That's the last part of that task above which I haven't done yet,
and for various reasons probably won't have time to try for a while.
Just so we don't forget it.

Chris

Christopher Kings-Lynne wrote:

-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)

Hey Bruce,

You probably should add 'Dump LOB comments in custom dump format' to the
todo. That's the last part of that task above which I haven't done yet,
and for various reasons probably won't have time to try for a while.
Just so we don't forget it.

Added:

* Dump large object comments in custom dump format

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073