programmatic way to fetch latest release for a given major.minor version

On 9 Apr 2007 14:47:20 -0700, Andrew Hammond <
andrew.george.hammond@gmail.com> wrote:

I'm writing a script that wants to know the latest release for a given
major.minor version. Is there some better way than parsing
http://www.postgresql.org/ftp/source/ or trying to connect to ftp
(which is invariably timing out on me today. Is that box getting
hammered or something?) and doing the parsing that? Both approaches
feel quite awkward to me.

Use wget to download via HTTP (added recently). Probably wise to add a
couple mirrors in your script.

On 4/9/07, CAJ CAJ <pguser@gmail.com> wrote:

On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
<andrew.george.hammond@gmail.com> wrote:

I'm writing a script that wants to know the latest release for a given
major.minor version. Is there some better way than parsing
http://www.postgresql.org/ftp/source/ or trying to

connect to ftp

(which is invariably timing out on me today. Is that box getting
hammered or something?) and doing the parsing that? Both approaches
feel quite awkward to me.

Use wget to download via HTTP (added recently). Probably wise to add a
couple mirrors in your script.

I'm not asking how to download stuff. I'm asking how to figure out the
current release number for a given major.minor. I thought that was
clear in my original post, but I guess not. For example, how do I
determine (programmatically) the lastest version of 8.1.

I'm also interested in a clever way to select one of the "close"
mirrors at random for downloading via http. However I had planned to
hold that question until I'd solved the first issue.

Andrew

Andrew Hammond escribi�:

On 4/9/07, CAJ CAJ <pguser@gmail.com> wrote:

On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
<andrew.george.hammond@gmail.com> wrote:

I'm writing a script that wants to know the latest release for a given
major.minor version. Is there some better way than parsing
http://www.postgresql.org/ftp/source/ or trying to

connect to ftp

(which is invariably timing out on me today. Is that box getting
hammered or something?) and doing the parsing that? Both approaches
feel quite awkward to me.

Use wget to download via HTTP (added recently). Probably wise to add a
couple mirrors in your script.

I'm not asking how to download stuff. I'm asking how to figure out the
current release number for a given major.minor. I thought that was
clear in my original post, but I guess not. For example, how do I
determine (programmatically) the lastest version of 8.1.

Maybe get configure.in from CVS:
cvs log configure.in
and parse the "symbolic names" list?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Here you go.

Fetches versions and prints most recent minor for each major
Tests all mirrors for speed and prints out the 4 fastest (takes some time)
http://www.crummy.com/software/BeautifulSoup/

Have a nice day !

#! /bin/env python
# -*- coding: utf-8 -*-

import urllib, BeautifulSoup, re, time, sys

def get_all_versions():
soup =
BeautifulSoup.BeautifulSoup( urllib.urlopen( "http://ftp3.fr.postgresql.org/pub/postgresql/source/&quot;
).read() )
for a in soup( 'a', {'href': re.compile( r"v\d+.\d+.\d+" ) } ):
yield map( int, re.search( r"v(\d+)\.(\d+)\.(\d+)*", a['href']
).groups() )

def get_latest_versions():
lastversions = {}
for a,b,c in sorted( get_all_versions() ):
lastversions[ (a,b) ] = c
return sorted( lastversions.items() )

def parse_query_string( url ):
return dict( map( urllib.unquote_plus, pair.split('=',1) ) for pair in
re.split( "&(?:amp;|)", urllib.splitquery( url )[1] ) )

def get_mirrors():
soup =
BeautifulSoup.BeautifulSoup( urllib.urlopen( "http://wwwmaster.postgresql.org/download/mirrors-ftp&quot;
).read() )
for a in soup( 'a', {'href': re.compile( r"\?setmir=.*url=" ) } ):
yield parse_query_string( a['href'] )['url']

def get_fastest_mirrors( urls, filename ):
for url in urls:
sys.stdout.write( " %s\r" % url )
t = time.time()
try:
urllib.urlopen( url + filename )
except:
pass
d = time.time()-t
print "%.02f s" % d
yield d, url

for major, minor in get_latest_versions():
print "%d.%d.%d" % (major[0], major[1], minor)

mirrors = get_mirrors()
fastest = sorted( get_fastest_mirrors( mirrors, "sync_timestamp" ))[:4]
for d, mirror in fastest:
print "%.02f s %s" % (d,mirror)

On Tue, 10 Apr 2007 00:34:02 +0200, Andrew Hammond
<andrew.george.hammond@gmail.com> wrote:

Andrew Hammond wrote:

On 4/9/07, CAJ CAJ <pguser@gmail.com> wrote:

On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
<andrew.george.hammond@gmail.com> wrote:

I'm writing a script that wants to know the latest release for a given
major.minor version. Is there some better way than parsing
http://www.postgresql.org/ftp/source/ or trying to

connect to ftp

(which is invariably timing out on me today. Is that box getting
hammered or something?) and doing the parsing that? Both approaches
feel quite awkward to me.

Use wget to download via HTTP (added recently). Probably wise to add a
couple mirrors in your script.

I'm not asking how to download stuff. I'm asking how to figure out the
current release number for a given major.minor. I thought that was
clear in my original post, but I guess not. For example, how do I
determine (programmatically) the lastest version of 8.1.

Currently you can't without parsing pages.

I'm also interested in a clever way to select one of the "close"
mirrors at random for downloading via http. However I had planned to
hold that question until I'd solved the first issue.

You might find this of use: http://www.postgresql.org/mirrors.xml. It's
updated regularly, so the mirrors should all have test OK within the
last 48 hours.

Regards, Dave.

On Tue, Apr 10, 2007 at 08:36:10AM +0100, Dave Page wrote:

Andrew Hammond wrote:

On 4/9/07, CAJ CAJ <pguser@gmail.com> wrote:

On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
<andrew.george.hammond@gmail.com> wrote:

I'm writing a script that wants to know the latest release for a given
major.minor version. Is there some better way than parsing
http://www.postgresql.org/ftp/source/ or trying to

connect to ftp

(which is invariably timing out on me today. Is that box getting
hammered or something?) and doing the parsing that? Both approaches
feel quite awkward to me.

Use wget to download via HTTP (added recently). Probably wise to add a
couple mirrors in your script.

I'm not asking how to download stuff. I'm asking how to figure out the
current release number for a given major.minor. I thought that was
clear in my original post, but I guess not. For example, how do I
determine (programmatically) the lastest version of 8.1.

Currently you can't without parsing pages.

I've been meaning to do this for a while now, and I guess this might be a
good motivator ;-) The idea would, of course, be to build this from a set of
common data with the frontpage of the website, so they're always in sync.
As I see it, there are basically three ways of doing it:

1) Fold it into mirrors.xml. That's going to be a bit hard given how the
schema is (since the root element is mirror), but it would work. Another
downside is that is that people who just want to check the latest version
will have to pull down the whole mirror list file. This file is also
rewritten often, so checks on If-Modified-Since will fail.

2) Create a new file with a specific schema. Something like:
<version>
<version major="8.2" minor="8.2.3" />
<version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

3) Use RSS format. Not exactly sure how it would look, but it can certainly
be done in RSS...

I think I prefer option 2, but what do others think?

//Magnus

Magnus Hagander wrote:

2) Create a new file with a specific schema. Something like:
<version>
<version major="8.2" minor="8.2.3" />
<version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

More like:

<versions>
<version major="8" minor="2" revision="3" />
<version major="8" minor="1" revision="8" />
</versions>

But I can't help thinking that we should have some additional values for
release notes, download sub-URLs (to be appended to the mirror roots) etc.

Regards, Dave.

I love Open Source XD

http://ethan.tira-thompson.com/cvslog2web/

Note that this is overkill (but it would look SEXY on the site).
However, the original poster probably wants to know when to update his
servers, so he won't care about CVS commits...
If there was a RSS feed of new postgres versions I'd subscribe to it
though.

On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:

Magnus Hagander wrote:

2) Create a new file with a specific schema. Something like:
<version>
<version major="8.2" minor="8.2.3" />
<version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

More like:

<versions>
<version major="8" minor="2" revision="3" />
<version major="8" minor="1" revision="8" />
</versions>

But that doesn't reflect our terminology. Per our own terminology, bioth
8.1 and 8.2 are major versions. Not only is that what we've been saying for
years, we even documented it at
http://www.postgresql.org/support/versioning.

But I can't help thinking that we should have some additional values for
release notes, download sub-URLs (to be appended to the mirror roots) etc.

Yeah, thus the X in XML :-)
But given that we might want to add things like this, having our own custom
XML format certainly makes a bit more sense, it might be harder to try to
trick it into RSS.

//Magnus

On Tue, Apr 10, 2007 at 11:09:50AM +0200, Listmail wrote:

I love Open Source XD

http://ethan.tira-thompson.com/cvslog2web/

Hmm. Don't tell people about my secret plans :)

(Though I hadn't looked at that piece of software in particylar)

Note that this is overkill (but it would look SEXY on the site).
However, the original poster probably wants to know when to update
his servers, so he won't care about CVS commits...
If there was a RSS feed of new postgres versions I'd subscribe to it
though.

That's the argument for RSS - you acn subscribe to it from standard RSS
readers.
Perhaps we need to do both...

//Magnus

Magnus Hagander wrote:

On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:

Magnus Hagander wrote:

2) Create a new file with a specific schema. Something like:
<version>
<version major="8.2" minor="8.2.3" />
<version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

More like:

<versions>
<version major="8" minor="2" revision="3" />
<version major="8" minor="1" revision="8" />
</versions>

But that doesn't reflect our terminology. Per our own terminology, bioth
8.1 and 8.2 are major versions. Not only is that what we've been saying for
years, we even documented it at
http://www.postgresql.org/support/versioning.

Yeah yeah, but terminology aside, having 2 or three digits in each
attribute is just wrong!

But I can't help thinking that we should have some additional values for
release notes, download sub-URLs (to be appended to the mirror roots) etc.

Yeah, thus the X in XML :-)

:-p

But given that we might want to add things like this, having our own custom
XML format certainly makes a bit more sense, it might be harder to try to
trick it into RSS.

Agreed.

/D

On Tue, Apr 10, 2007 at 12:03:44PM +0100, Dave Page wrote:

Magnus Hagander wrote:

On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:

Magnus Hagander wrote:

2) Create a new file with a specific schema. Something like:
<version>
<version major="8.2" minor="8.2.3" />
<version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

More like:

<versions>
<version major="8" minor="2" revision="3" />
<version major="8" minor="1" revision="8" />
</versions>

But that doesn't reflect our terminology. Per our own terminology, bioth
8.1 and 8.2 are major versions. Not only is that what we've been saying for
years, we even documented it at
http://www.postgresql.org/support/versioning.

Yeah yeah, but terminology aside, having 2 or three digits in each
attribute is just wrong!

Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
sense to say you're on version 8, in the given context, so why should the
XML data pretend there is?

//Magnus

Magnus Hagander wrote:

Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
sense to say you're on version 8, in the given context, so why should the
XML data pretend there is?

Because serving the data in the decomposed format gives the consumer the
maximum flexibility to do as they wish with the data. I find it hard to
see why, as a relational database guy, you'd want to offer the data as
"8.1", "8.1.3" etc. when you can just give the three parts separately
and allow people to check whatever they need without having to chop up
strings!

Imagine wanting to display only the details of the 8.x releases on a
site for example. In your schema, you'd have to use a substring match on
an attribute value to filter out 6.x and 7.x.

Regards,D ave.

Yeah yeah, but terminology aside, having 2 or three digits in each
attribute is just wrong!

Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
sense to say you're on version 8, in the given context, so why should the
XML data pretend there is?

//Magnus

Just pretend that :

- version = a tuple of integers (a, b, c, ...)
- major = (a, b)
- minor = (c, ...)

Besides, that is sortable (unlike strings where 15 < 2) :

latest minor for major :
major, max(minor) where major = what you want

<pgversion><major><int value="8" /><int value="2" /></major><minor><int
value="3" /></minor></pgversion>

from BeautifulSoup import BeautifulSoup as Soup
s = Soup("""<pgversion><major><int value="8" /><int value="2"
/></major><minor><int value="3" /></minor></pgversion>""" )

v = s.find('pgversion')
[int(x['value']) for x in v.find('major') ]

[8, 2]

[int(x['value']) for x in v.find('minor') ]

[3]:

On Tue, Apr 10, 2007 at 12:35:38PM +0100, Dave Page wrote:

Magnus Hagander wrote:

Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
sense to say you're on version 8, in the given context, so why should the
XML data pretend there is?

Because serving the data in the decomposed format gives the consumer the
maximum flexibility to do as they wish with the data. I find it hard to
see why, as a relational database guy, you'd want to offer the data as
"8.1", "8.1.3" etc. when you can just give the three parts separately
and allow people to check whatever they need without having to chop up
strings!

Imagine wanting to display only the details of the 8.x releases on a
site for example. In your schema, you'd have to use a substring match on
an attribute value to filter out 6.x and 7.x.

That is actually precisely my point. It makes *no sense* to filter based on
8.x. 8.0 is no more a major release than 7.4. And we'd encourage people to
do that.

//Magnus

On Tue, Apr 10, 2007 at 12:18:52PM +0200, Magnus Hagander wrote:

But given that we might want to add things like this, having our own custom
XML format certainly makes a bit more sense, it might be harder to try to
trick it into RSS.

I'd say do it in the format you're most comfortable with. Then some
XSLT wizard can convert it into RSS from there...

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Tue, Apr 10, 2007 at 02:44:42PM +0200, Martijn van Oosterhout wrote:

On Tue, Apr 10, 2007 at 12:18:52PM +0200, Magnus Hagander wrote:

But given that we might want to add things like this, having our own custom
XML format certainly makes a bit more sense, it might be harder to try to
trick it into RSS.

I'd say do it in the format you're most comfortable with. Then some
XSLT wizard can convert it into RSS from there...

Too late, I already did the RSS. Which was very simple, because of the RSS
framework stuff already in place on the website.

*However*, that one doesn't contain all the information that a program
would want, and certainly not in an easily parseable format for them. It's
designed for people in the other end. So I intend to do a "proper XML
format" as well.

//Magnus

Am Dienstag, 10. April 2007 13:35 schrieb Dave Page:

Imagine wanting to display only the details of the 8.x releases on a
site for example. In your schema, you'd have to use a substring match on
an attribute value to filter out 6.x and 7.x.

That use case is just as valid as wanting to show only releases >= 7.4. The
user could run an arithmetic expression to check for that and >= 8.0.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Magnus Hagander wrote:

That is actually precisely my point. It makes *no sense* to filter based on
8.x. 8.0 is no more a major release than 7.4.

Yes it is - that's precisely why it was 8.0 and not 7.5.

/D