pg_restore -j <nothing>

Alvaro Herrera wrote:

Hi,

I just noticed (!) that Make accepts an argument-less -j option, which
it takes to mean "use as many parallel jobs as possible". As far as I
see in our pg_restore code, we don't even accept an argumentless -j
option; was this deviation from the Make precedent on purpose, or were
we just not following Make at all on this?

I have to admit that I'm not really sure whether this kind of usage
would be a reasonable thing for pg_restore to support.

(Even if this was a good idea, I'm not suggesting that it be implemented
for 8.4. But if it is, then maybe it deserves a TODO entry.)

An unlimited pg_restore -j seems pretty scary.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Alvaro Herrera wrote:

Hi,

I just noticed (!) that Make accepts an argument-less -j option, which
it takes to mean "use as many parallel jobs as possible". As far as I
see in our pg_restore code, we don't even accept an argumentless -j
option; was this deviation from the Make precedent on purpose, or were
we just not following Make at all on this?

I have to admit that I'm not really sure whether this kind of usage
would be a reasonable thing for pg_restore to support.

(Even if this was a good idea, I'm not suggesting that it be implemented
for 8.4. But if it is, then maybe it deserves a TODO entry.)

Thoughts?

There was no intention to follow Make.

And I think it's far far too early to be planning "improvements" of this
kind. We need to see how it gets used in the field.

cheers

andrew

Bruce Momjian <bruce@momjian.us> writes:

Alvaro Herrera wrote:

I just noticed (!) that Make accepts an argument-less -j option, which
it takes to mean "use as many parallel jobs as possible".

An unlimited pg_restore -j seems pretty scary.

Yeah. Even if Make has a sane way to estimate how many jobs it should
use, I'm not sure that pg_restore does. (The most obvious heuristic
for Make is to try to find out how many CPUs there are --- but at
least it's running on the same machine it's going to be eating CPU
on. pg_restore can't assume that.)

regards, tom lane

On Thursday 23 April 2009 01:26:04 Alvaro Herrera wrote:

I just noticed (!) that Make accepts an argument-less -j option, which
it takes to mean "use as many parallel jobs as possible". As far as I
see in our pg_restore code, we don't even accept an argumentless -j
option; was this deviation from the Make precedent on purpose, or were
we just not following Make at all on this?

There was likely no strong intention to follow make on this. A small problem
would be that getopt doesn't portably support single-letter options with
optional arguments.

The main problem, however, is that make -j is pretty useless and dangerous.
Using it on a large parallel-make-safe project can easily lock up (thrash)
your machine. make -j together with -l (--load-average) is kind of useful, I
guess, but exactly how "load average" translated to a PostgreSQL database
system is to be determined.

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

Yeah. Even if Make has a sane way to estimate how many jobs it should
use, I'm not sure that pg_restore does. (The most obvious heuristic
for Make is to try to find out how many CPUs there are --- but at
least it's running on the same machine it's going to be eating CPU
on. pg_restore can't assume that.)

I'm not sure if I'd consider it 'sane', but make basically uses the
dependency information, if a job can be run based on its dependency
requirements, then it's started. For small projects, this isn't
necessairly terrible, but it's not something I would generally
recommend.

I don't see any reasonable implementation, or justification, for
supporting something like that in pg_restore.

Thanks,

Stephen