initdb

Andrew Dunstan <andrew@dunslane.net> writes:

I now have a C implementation of initdb, ...
There's a little work still to go (see below), but I'd appreciate some
eyeballs on it to see if I have made any major booboos, or could have
done things better. What's the best way to proceed?

Put it on a web page and post the link, or if you don't have a website,
send it to pgsql-patches (clearly marked as "not ready to apply").

regards, tom lane

Excellent idea.

Here's the URL: http://www.dunslane.net/~andrew/Initdb_In_C.html

cheers

andrew

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>

I have added this URL to the Win32 page, and removed the "initdb C" TODO
item. Great job. Do you wantt this in Win32 CVS or should we just wait
for 7.5 to start?

---------------------------------------------------------------------------

Andrew Dunstan wrote:

Excellent idea.

Here's the URL: http://www.dunslane.net/~andrew/Initdb_In_C.html

cheers

andrew

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>

Andrew Dunstan <andrew@dunslane.net> writes:

I now have a C implementation of initdb, ...
There's a little work still to go (see below), but I'd appreciate some
eyeballs on it to see if I have made any major booboos, or could have
done things better. What's the best way to proceed?

Put it on a web page and post the link, or if you don't have a website,
send it to pgsql-patches (clearly marked as "not ready to apply").

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

I'm for putting it in Win32... my 2c.

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Friday, 10 October 2003 11:39 AM
To: Andrew Dunstan
Cc: Postgresql Hackers; pgsql-hackers-win32
Subject: Re: [pgsql-hackers-win32] [HACKERS] initdb

I have added this URL to the Win32 page, and removed the
"initdb C" TODO
item. Great job. Do you wantt this in Win32 CVS or should
we just wait
for 7.5 to start?

--------------------------------------------------------------
-------------

Andrew Dunstan wrote:

Excellent idea.

Here's the URL: http://www.dunslane.net/~andrew/Initdb_In_C.html

cheers

andrew

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>

Andrew Dunstan <andrew@dunslane.net> writes:

I now have a C implementation of initdb, ...
There's a little work still to go (see below), but I'd

appreciate some

eyeballs on it to see if I have made any major booboos,

or could have

done things better. What's the best way to proceed?

Put it on a web page and post the link, or if you don't

have a website,

send it to pgsql-patches (clearly marked as "not ready to apply").

---------------------------(end of

broadcast)---------------------------

TIP 6: Have you searched our list archives?

http://archives.postgresql.org

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, 
Pennsylvania 19073

---------------------------(end of
broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

--- 
WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS: 
THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041 
TEL: +61 2 9555 1544 FAX: +61 2 9555 6911 
Certain disclaimers and policies apply to all email sent from Memetrics.
For the full text of these disclaimers and policies see 
http://www.memetrics.com/emailpolicy.html

There will be a new version out there soon. When I am happy with it I will
let you know - right now I am dealing with 2 issues - setlocale(LC_ALL,"")
doesn't read from the environment on Windows, and the program is possibly
not picking up the buffers/connections properly.

Please don't put the version currently out there in CVS - the files won't
even be the same - initdb-scripts.h will disappear and there will be a new
file called "system_views.sql".

There is no urgency about this - until we have a working postgres executable
on Windows initdb is useless anyway. My aim has been to have initdb ready
when postgres is ready. Of course, I test against Unix all the time to make
sure nothing gets broken.

cheers

andrew

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Andrew Dunstan" <andrew@dunslane.net>
Cc: "Postgresql Hackers" <pgsql-hackers@postgresql.org>;
"pgsql-hackers-win32" <pgsql-hackers-win32@postgresql.org>
Sent: Thursday, October 09, 2003 9:39 PM
Subject: Re: [pgsql-hackers-win32] [HACKERS] initdb

I have added this URL to the Win32 page, and removed the "initdb C" TODO
item. Great job. Do you wantt this in Win32 CVS or should we just wait
for 7.5 to start?

--------------------------------------------------------------------------

-

Andrew Dunstan wrote:

Excellent idea.

Here's the URL: http://www.dunslane.net/~andrew/Initdb_In_C.html

cheers

andrew

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>

Andrew Dunstan <andrew@dunslane.net> writes:

I now have a C implementation of initdb, ...
There's a little work still to go (see below), but I'd appreciate

some

eyeballs on it to see if I have made any major booboos, or could

have

done things better. What's the best way to proceed?

Put it on a web page and post the link, or if you don't have a

website,

send it to pgsql-patches (clearly marked as "not ready to apply").

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania

19073

Andrew Dunstan wrote:

There will be a new version out there soon. When I am happy with it I will
let you know - right now I am dealing with 2 issues - setlocale(LC_ALL,"")
doesn't read from the environment on Windows, and the program is possibly
not picking up the buffers/connections properly.

Please don't put the version currently out there in CVS - the files won't
even be the same - initdb-scripts.h will disappear and there will be a new
file called "system_views.sql".

There is no urgency about this - until we have a working postgres executable
on Windows initdb is useless anyway. My aim has been to have initdb ready
when postgres is ready. Of course, I test against Unix all the time to make
sure nothing gets broken.

Fine. We have the URL on the win32 web site, and that's all we need.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

There's a new version available at
http://www.dunslane.net/~andrew/Initdb_In_C.html - I think it takes care of
all of Peter's comments. I removed initdb-scripts.h - the smaller scripts
were moved into initdb.c with SQL comments turned into C comments, and the
large script to set up system views becomes its own file, which is now
passed to postgres with single line mode turned off.

I have just realised that I seem to have skipped the bki file version check.
I will fix that in the next version.

cheers

andrew

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Andrew Dunstan" <andrew@dunslane.net>
Cc: "Postgresql Hackers" <pgsql-hackers@postgresql.org>;
"pgsql-hackers-win32" <pgsql-hackers-win32@postgresql.org>
Sent: Thursday, October 09, 2003 10:07 PM
Subject: Re: [pgsql-hackers-win32] [HACKERS] initdb

Andrew Dunstan wrote:

There will be a new version out there soon. When I am happy with it I

will

let you know - right now I am dealing with 2 issues -

setlocale(LC_ALL,"")

doesn't read from the environment on Windows, and the program is

possibly

not picking up the buffers/connections properly.

Please don't put the version currently out there in CVS - the files

won't

even be the same - initdb-scripts.h will disappear and there will be a

new

file called "system_views.sql".

There is no urgency about this - until we have a working postgres

executable

on Windows initdb is useless anyway. My aim has been to have initdb

ready

when postgres is ready. Of course, I test against Unix all the time to

make

sure nothing gets broken.

Fine. We have the URL on the win32 web site, and that's all we need.

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania

19073

Bruce,

I think this is now ready to import into the WIN32 branch. I have ironed out
the cygwin dependencies, and posted a note about what we'll need to do
inside postgres to handle the environment values for setlocale() on Windows.

If you want to rip out the unlink stuff (see previous email thread) we'll
need a version of dirmod.c that is compiled with -DFRONTEND - I guess libpq
is the obvious place for that.

Apart from initdb.c we'll need the file system_views.sql, which can be taken
without change (unless someone want to add comments to it), and Makefile
modifications (see my Makefile on the web for an example, but it probably
doesn't comply with our standard way of doing things).

Anything broken can then be fixed with patches - I have about 7 versions on
5 machines now and my head is starting to swim -).

pg_id and pg_encoding now become redundant in their entirety, as does
initdb.sh.

cheers

andrew

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Andrew Dunstan" <andrew@dunslane.net>
Cc: "Postgresql Hackers" <pgsql-hackers@postgresql.org>;
"pgsql-hackers-win32" <pgsql-hackers-win32@postgresql.org>
Sent: Thursday, October 09, 2003 10:07 PM
Subject: Re: [pgsql-hackers-win32] [HACKERS] initdb

Andrew Dunstan wrote:

There will be a new version out there soon. When I am happy with it I

will

let you know - right now I am dealing with 2 issues -

setlocale(LC_ALL,"")

doesn't read from the environment on Windows, and the program is

possibly

not picking up the buffers/connections properly.

Please don't put the version currently out there in CVS - the files

won't

even be the same - initdb-scripts.h will disappear and there will be a

new

file called "system_views.sql".

There is no urgency about this - until we have a working postgres

executable

on Windows initdb is useless anyway. My aim has been to have initdb

ready

when postgres is ready. Of course, I test against Unix all the time to

make

sure nothing gets broken.

Fine. We have the URL on the win32 web site, and that's all we need.

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania

19073

Andrew Dunstan wrote:

Bruce,

I think this is now ready to import into the WIN32 branch. I have ironed out
the cygwin dependencies, and posted a note about what we'll need to do
inside postgres to handle the environment values for setlocale() on Windows.

If you want to rip out the unlink stuff (see previous email thread) we'll
need a version of dirmod.c that is compiled with -DFRONTEND - I guess libpq
is the obvious place for that.

Apart from initdb.c we'll need the file system_views.sql, which can be taken
without change (unless someone want to add comments to it), and Makefile
modifications (see my Makefile on the web for an example, but it probably
doesn't comply with our standard way of doing things).

Anything broken can then be fixed with patches - I have about 7 versions on
5 machines now and my head is starting to swim -).

pg_id and pg_encoding now become redundant in their entirety, as does
initdb.sh.

Here is a slightly modified version of Andrew's great work in making a C
version of initdb. Other than minor cleanups, the only big change was
to remove rmdir handling because we using rm -r and rmdir /s in
commands/dbcommands.c, so we might as use the same thing for initdb.c
rather than having code that traverses the directory tree doing 'rm'.

The other change was to remove the unlink code and instead use
port/dirmod.c's version.

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I would like to apply this in the next few days to HEAD before initdb.sh
drifts. We are no longer using the WIN32_DEV branch because all our
work is now in 7.5 head.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I had some concern about whether Andrew's rewrite had tracked all the
recent changes to the initdb shell-script sources. Are you both
confident that it is up to date?

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I had some concern about whether Andrew's rewrite had tracked all the
recent changes to the initdb shell-script sources. Are you both
confident that it is up to date?

Yes. Bruce has picked up the one change I didn't track (revision 1.204).

cheers

andrew

Bruce Momjian wrote:

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I would like to apply this in the next few days to HEAD before initdb.sh
drifts. We are no longer using the WIN32_DEV branch because all our
work is now in 7.5 head.

I will double check this over the weekend.

cheers

andrew

Bruce Momjian wrote:

Here is a slightly modified version of Andrew's great work in making a C
version of initdb. Other than minor cleanups, the only big change was
to remove rmdir handling because we using rm -r and rmdir /s in
commands/dbcommands.c, so we might as use the same thing for initdb.c
rather than having code that traverses the directory tree doing 'rm'.

The other change was to remove the unlink code and instead use
port/dirmod.c's version.

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I would like to apply this in the next few days to HEAD before initdb.sh
drifts. We are no longer using the WIN32_DEV branch because all our
work is now in 7.5 head.

My comments:

I have no problem with shelling out to rmdir - although my goal was to
avoid shelling out to anything other than postgres itself. I think
recreating the datadir if we didn't create it initially should be OK in
that case, and it makes the code simpler. Not handling dot files in the
Unix case should also be fine, as we know the directory was empty to
start with and we don't create any.

Regarding the #endif comments you removed - Peter had asked me to put
them in. See
http://archives.postgresql.org/pgsql-hackers/2003-10/msg00352.php

You put a comment on canonicalise_path() asking if it is needed. The
short answer is very much "yes". The Windows command processor will
accept suitably quoted paths with forward slashes, but barfs badly with
mixed forward and back slashes. (See recent discussion on
hackers-win32).Removing the trailing slash on a path means we never get
ugly double slashes. Feel free to put that info in as a comment if you
think it is needed.

The changes you made for the final message are not going to work for
Windows - if pgpath or pg_data contain spaces we need quotes (even on
Unix, although there most people know better than to put spaces in
names). Also see above about mixed slashes. Also, putting a \ before
pg_data will be nasty if it starts with a drive or network specifier -
or even without (you might turn a directory specifier into a network
drive specifier). It's just a message, though, and we shouldn't hold up
for that - we can patch it to get it right.

(Getting the path and slash thing working portably was by far the
biggest headache in this project - the rest was just grunt work for the
most part, although I learned a few interesting things.)

Otherwise I'm fine with this.

cheers

andrew

Andrew Dunstan wrote:

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I had some concern about whether Andrew's rewrite had tracked all the
recent changes to the initdb shell-script sources. Are you both
confident that it is up to date?

Yes. Bruce has picked up the one change I didn't track (revision 1.204).

Yes, I was concerned too that everything was in there. I checked the
initdb.sh logs and found that the only thing not added was the checking
of the max number of connections before checking the max number of
buffers, which I added. The other stuff was in there. I also checked
pg_id's recent changes and those were in there too.

Andrew, I assume this was a new implementation of initdb, and not taken
from an older initdb C implementation made by a company.

This isn't really a patch but a C replacement of a critical shell
script so there is reason to double-check things.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Yes, I was concerned too that everything was in there. I checked the
initdb.sh logs and found that the only thing not added was the checking
of the max number of connections before checking the max number of
buffers, which I added. The other stuff was in there. I also checked
pg_id's recent changes and those were in there too.

Andrew, I assume this was a new implementation of initdb, and not taken
from an older initdb C implementation made by a company.

This isn't really a patch but a C replacement of a critical shell
script so there is reason to double-check things.

Yes, I worked from initdb.sh, not from any other source. It's "all my
own work" :-) I think I started with 1.201 and later upgraded to 1.203.

I agree it needs careful checking - the more eyeballs the better.

cheers

andrew

Andrew Dunstan wrote:

Bruce Momjian wrote:

Here is a slightly modified version of Andrew's great work in making a C
version of initdb. Other than minor cleanups, the only big change was
to remove rmdir handling because we using rm -r and rmdir /s in
commands/dbcommands.c, so we might as use the same thing for initdb.c
rather than having code that traverses the directory tree doing 'rm'.

The other change was to remove the unlink code and instead use
port/dirmod.c's version.

It passes all the regression tests. I have also included a diff against
Andrew's version so you can see my changes. It seems Andrew had a very
current version of initdb. The only update he missed was the change to
test the number of connections before shared buffers --- I made that
change myself.

I would like to apply this in the next few days to HEAD before initdb.sh
drifts. We are no longer using the WIN32_DEV branch because all our
work is now in 7.5 head.

My comments:

I have no problem with shelling out to rmdir - although my goal was to
avoid shelling out to anything other than postgres itself. I think
recreating the datadir if we didn't create it initially should be OK in
that case, and it makes the code simpler. Not handling dot files in the
Unix case should also be fine, as we know the directory was empty to
start with and we don't create any.

Good. I can do rmdir() in C in port/dirmod.c if we need it. Right now
we are doing system(rm/rmdir) in dbcommands.c so we should consistent.
Let's stay with system(rm/rmdir) and if it doesn't work as we expect, we
can add your rmdir() code and put it in port/dirmod.c.

Regarding the #endif comments you removed - Peter had asked me to put
them in. See
http://archives.postgresql.org/pgsql-hackers/2003-10/msg00352.php

Peter, I thought we add:

#endif /* port... */

only when the define covers many lines of code. This case:

#ifdef WIN32
some code
#endif

seems clearer than:

#ifdef WIN32
some code
#endif /* WIN32 */

Of course, if the code block is very large, a closing comment can
sometimes help. Personally, I don't like the comments anytime, but if
people like them on long blocks, I can understand that.

You put a comment on canonicalise_path() asking if it is needed. The
short answer is very much "yes". The Windows command processor will
accept suitably quoted paths with forward slashes, but barfs badly with
mixed forward and back slashes. (See recent discussion on
hackers-win32).Removing the trailing slash on a path means we never get
ugly double slashes. Feel free to put that info in as a comment if you
think it is needed.

Done.

The changes you made for the final message are not going to work for
Windows - if pgpath or pg_data contain spaces we need quotes (even on
Unix, although there most people know better than to put spaces in
names). Also see above about mixed slashes. Also, putting a \ before
pg_data will be nasty if it starts with a drive or network specifier -
or even without (you might turn a directory specifier into a network
drive specifier). It's just a message, though, and we shouldn't hold up
for that - we can patch it to get it right.

I am confused. The only change I made was to have the quotes appear
only on WIN32.

#ifndef WIN32
#define QUOTE_PATH ""
#else
#define QUOTE_PATH "\""
#endif

...

printf("\n"
"Success. You can now start the database server using:\n\n"
" %s/postmaster -D %s%s%s\n"
"or\n"
" %s/pg_ctl -D %s%s%s -l logfile start\n\n",
pgpath, QUOTE_PATH, pg_data, QUOTE_PATH,
pgpath, QUOTE_PATH, pg_data, QUOTE_PATH);

(I also merged multiple \n into a single line.) My logic was that
spaces in directory names are much more common on WIN32, so we should
display the quotes only on WIN32. Perhaps you read "\"" as "\\"?

(Getting the path and slash thing working portably was by far the
biggest headache in this project - the rest was just grunt work for the
most part, although I learned a few interesting things.)

Otherwise I'm fine with this.

Thanks for reviewing my work. I am surprised how small the new initdb.c
is. It is 50k vs 38k for initdb.sh. Of course, system_views.sql is
another 10k, but still, it is smaller than I thought, and quite clear.

Thanks.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Andrew Dunstan wrote:

Bruce Momjian wrote:

Yes, I was concerned too that everything was in there. I checked the
initdb.sh logs and found that the only thing not added was the checking
of the max number of connections before checking the max number of
buffers, which I added. The other stuff was in there. I also checked
pg_id's recent changes and those were in there too.

Andrew, I assume this was a new implementation of initdb, and not taken
from an older initdb C implementation made by a company.

This isn't really a patch but a C replacement of a critical shell
script so there is reason to double-check things.

Yes, I worked from initdb.sh, not from any other source. It's "all my
own work" :-) I think I started with 1.201 and later upgraded to 1.203.

I agree it needs careful checking - the more eyeballs the better.

The great part is that it look so much like our code, unlike the
commerical port code I have seen for initdb in the past. This certainly
moves us forward on Win32.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Andrew Dunstan writes:

recreating the datadir if we didn't create it initially should be OK in
that case, and it makes the code simpler.

That should be avoided, because you'll have trouble recreating the
original directory with all its properties such as ownership, permissions,
etc., at least not without making the code anything but simpler. There
might even be a situation were you are allowed to delete the directory but
cannot create a new one.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

Andrew Dunstan writes:

recreating the datadir if we didn't create it initially should be OK in
that case, and it makes the code simpler.

That should be avoided, because you'll have trouble recreating the
original directory with all its properties such as ownership, permissions,
etc., at least not without making the code anything but simpler. There
might even be a situation were you are allowed to delete the directory but
cannot create a new one.

Consider also the strong likelihood that the data directory's parent
directory is owned by root, so that you do not have the ability to
delete and recreate the data directory because you don't have write
permission on its parent. The main reason initdb is set up to be able
to start with an existing-but-empty data dir is exactly because creating
that directory may have required permissions that initdb itself hasn't
got.

regards, tom lane