tinterval - operator problems on AIX

On AIX timestamp and horology regression fails in current, because
timestamp - interval for result timestamps that are before 1970 (epoch ?)
are off by one hour. I think this is not an issue for an adapted expected file,
but a new (in 7.1beta) bug. But I am at no means an expert at what the result
should be when substracting 34 years from epoch or 'Mon Dec 30 17:32:01 1996 PST'.

Hi Andreas. It is certainly true that the behavior has changed for the
new release, but afaik it can not be put into the "bug" category.

The difference is that, before, I rotated date/time into the correct
time zone for arithmetic by adding/subtracting the current time zone.
For (date/time)-(interval), this resulted in evaluating the result in
the same time zone as the starting date/time, which was not correct.

The time zone is now evaluated in the time zone of the result, rather
than the input, using system support routines from libc. But that will
expose troubles on some platforms with time zone support before 1970.
Some platforms are worse than others; my experience has been that
zinc-based systems such as Linux, FreeBSD, and Tru64 are pretty good,
Suns are the best, and, well, apparently AIX is not ;)

- Thomas

On AIX timestamp and horology regression fails in current, because
timestamp - interval for result timestamps that are before

1970 (epoch ?)

are off by one hour. I think this is not an issue for an

adapted expected file,

but a new (in 7.1beta) bug. But I am at no means an expert

at what the result

should be when substracting 34 years from epoch or 'Mon Dec

30 17:32:01 1996 PST'.

Hi Andreas. It is certainly true that the behavior has changed for the
new release, but afaik it can not be put into the "bug" category.

The difference is that, before, I rotated date/time into the correct
time zone for arithmetic by adding/subtracting the current time zone.
For (date/time)-(interval), this resulted in evaluating the result in
the same time zone as the starting date/time, which was not correct.

The time zone is now evaluated in the time zone of the result, rather
than the input, using system support routines from libc. But that will
expose troubles on some platforms with time zone support before 1970.
Some platforms are worse than others; my experience has been that
zinc-based systems such as Linux, FreeBSD, and Tru64 are pretty good,
Suns are the best, and, well, apparently AIX is not ;)

What I do not understand is, that I thought a time in december cannot be
affected by such math because it is clearly not in the daylight savings
months of the year ? Also I thought that you recognize daylight savings time
by the PDT instead of PST, and PostgreSQL shows a result with PST as I
would have expected, but one hour off.

e.g.: Mon Dec 30 17:32:01 1996 PST - @ 34 years --> Sun Dec 30 16:32:01 1962 PST

How do we go about resolving this ? I hesitate to mark a bogus result
"ok".

Andreas

What I do not understand is, that I thought a time in december cannot be
affected by such math because it is clearly not in the daylight savings
months of the year ? Also I thought that you recognize daylight savings time
by the PDT instead of PST, and PostgreSQL shows a result with PST as I
would have expected, but one hour off.

By the time the math is actually done, the date is in GMT. So the issue
is: how does one assign the time zone for the result? Or more properly,
how does one account for time zone shifts (e.g. Daylight time vs
Standard time) when doing date/time arithmetic? The previous scheme
rotated the date/time by the same amount as the original date input. The
current scheme uses your system's date/time support functions to deduce
the correct time zone and DST setting once the math is complete. In
principle, this gives your system the opportunity to come up with the
correct answer, where previously the answer was not what might be
expected for any platform.

e.g.: Mon Dec 30 17:32:01 1996 PST - @ 34 years --> Sun Dec 30 16:32:01 1962 PST
How do we go about resolving this ? I hesitate to mark a bogus result
"ok".

The result must be what the AIX developers want ;)

I've made some assumptions in this conversation:

1) Some platforms, Linux included, give "correct results". I think I
checked this thoroughly, but, well, you know how that goes sometimes...

2) You are diff'ing the current "expected" against current "results",
and not a previous version of "expected".

3) Some platforms do not do the right thing wrt time zones for dates
before 1970. For example, the USA had some weird time zones during WWII,
and this is reflected in Sun's time zone database but not quite as
accurately in the open source zinc package that Linux and others use.
And other platforms (such as AIX?) prefer to do it themselves and get it
wrong altogether.

- Thomas

On Tue, Jan 09, 2001 at 05:05:34PM +0000, Thomas Lockhart wrote:

What I do not understand is, that I thought a time in december cannot be
affected by such math because it is clearly not in the daylight savings
months of the year ? Also I thought that you recognize daylight savings
time by the PDT instead of PST, and PostgreSQL shows a result with PST
as I would have expected, but one hour off.

By the time the math is actually done, the date is in GMT. So the issue
is: how does one assign the time zone for the result?
...
3) Some platforms do not do the right thing wrt time zones for dates
before 1970. For example, the USA had some weird time zones during WWII,
and this is reflected in Sun's time zone database but not quite as
accurately in the open source zinc package that Linux and others use.
And other platforms (such as AIX?) prefer to do it themselves and get it
wrong altogether.

Indeed, Unix time_t is not required to represent any time before 1970,
so it may be that on AIX the time zone computation is treating it as
a very large unsigned (therefore positive) value, coming up with a time
in the summer of (perhaps) 2098, and adjusting the time accordingly;
then feeding the time to a different conversion that treats negative
time_t values as really negative.

Of course, it is a grievous error to use Unix time_t to represent
real-world dates (e.g. birthdates). Didn't Y2K teach us anything?
ISTM the correct response is to remove test cases from the regression
suite that refer to times before 1970 and after 2038, if they involve
a time_t representation.

Nathan Myers
ncm@zembu.com

The time zone is now evaluated in the time zone of the result, rather
than the input, using system support routines from libc.

Ok, I have the answer.

On AIX mktime(3) leaves tm_isdst at -1 if it does not have timezone
info for that particular year and returns -1.

From man page:

The mktime subroutine returns the specified time in seconds encoded as a value of
type time_t. If the time cannot be represented, the function returns the value
(time_t)-1.

The following code then makes savings time out of the -1.
tz = (tm->tm_isdst ? (timezone - 3600) : timezone);

What now ?

Andreas

On AIX mktime(3) leaves tm_isdst at -1 if it does not have timezone
info for that particular year and returns -1.
The following code then makes savings time out of the -1.
tz = (tm->tm_isdst ? (timezone - 3600) : timezone);

Hmm. That description is consistant with what I see in the Linux man
page. So I should check for (tm->tm_isdst > 0) rather than checking for
non-zero?

Would you like to test that on your machine? I'll try it here, and if
successful will consider this a bug report and a necessary fix for 7.1.

It is interesting that this is the only place in all of our code which
tickles this bug; afaik this line of code appears in other places too.
Can you find a case where the current code fails when you are not doing
arithmetic? Perhaps there are some more tests we could include in the
regression tests??

It is also interesting that afaik AIX is the only machine exhibiting
this problem. The before-1970 range of dates is known to occur in some
use cases, and having *something* in the timezone database isn't that
difficult :/

- Thomas

Thomas Lockhart writes:

On AIX mktime(3) leaves tm_isdst at -1 if it does not have timezone
info for that particular year and returns -1.
The following code then makes savings time out of the -1.
tz = (tm->tm_isdst ? (timezone - 3600) : timezone);

Hmm. That description is consistant with what I see in the Linux
man page. So I should check for (tm->tm_isdst > 0) rather than
checking for non-zero?

Would you like to test that on your machine? I'll try it here, and
if successful will consider this a bug report and a necessary fix
for 7.1.

I have machines running AIX 4.1.5, 4.2.1, and 4.3.3 if you would like
to send me your test programs.
--
Pete Forman -./\.- Disclaimer: This post is originated
WesternGeco -./\.- by myself and does not represent
pete.forman@westerngeco.com -./\.- opinion of Schlumberger, Baker
http://www.crosswinds.net/~petef -./\.- Hughes or their divisions.

On AIX mktime(3) leaves tm_isdst at -1 if it does not have timezone
info for that particular year and returns -1.
The following code then makes savings time out of the -1.
tz = (tm->tm_isdst ? (timezone - 3600) : timezone);

Hmm. That description is consistant with what I see in the Linux man
page. So I should check for (tm->tm_isdst > 0) rather than
checking for non-zero?

It is obviously not possible to determine tm_isdst with mktime for a
negative time_t. Thus with above fix PST works, but PDT is then busted :-(
localtime does convert a negative time_t correctly including dst.
Is there another way to determine tm_isdst ?

Andreas

On AIX mktime(3) leaves tm_isdst at -1 if it does not have timezone
info for that particular year and returns -1.
The following code then makes savings time out of the -1.
tz = (tm->tm_isdst ? (timezone - 3600) : timezone);

Hmm. That description is consistant with what I see in the Linux man
page. So I should check for (tm->tm_isdst > 0) rather than
checking for non-zero?

It is obviously not possible to determine tm_isdst with mktime for a
negative time_t. Thus with above fix PST works, but PDT is then busted :-(

Obvious to AIX only? My conclusion is that the AIX timezone database is
damaged or missing for pre-1970 dates, but that other systems bothered
to get it at least somewhat right. Is there another issue here that I'm
missing?

localtime does convert a negative time_t correctly including dst.
Is there another way to determine tm_isdst ?

Yes. Replace AIX with Linux or something else, then recompile Postgres
;)

Seriously, not that I know of. The problem is that there is no API for
accessing time zone info other than the localtime()/mktime() kinds of
calls, so we are stuck using that (or stuck repackaging something like
zinc into Postgres directly, which afaik is not anything we would be
interested in doing).

- Thomas

On AIX mktime(3) leaves tm_isdst at -1 if it does not have timezone
info for that particular year and returns -1.
The following code then makes savings time out of the -1.
tz = (tm->tm_isdst ? (timezone - 3600) : timezone);

Hmm. That description is consistant with what I see in the Linux man
page. So I should check for (tm->tm_isdst > 0) rather than
checking for non-zero?

It is obviously not possible to determine tm_isdst with mktime for a
negative time_t. Thus with above fix PST works, but PDT is then busted :-(

Obvious to AIX only?

Yes. The whole subject only concerns AIX (at least so far).

My conclusion is that the AIX timezone database is
damaged or missing for pre-1970 dates, but that other systems bothered
to get it at least somewhat right. Is there another issue here that I'm
missing?

The tz db is neighter damaged nor missing anything (see below). Only mktime
does not work for some (maybe even avoidable) reason for dates before 1970.

localtime does convert a negative time_t correctly including dst.
Is there another way to determine tm_isdst ?

Yes. Replace AIX with Linux or something else, then recompile Postgres
;)

As I see it, the Linux results are also not 100 % correct in respect to dates
before 1970. (based on assumption that Solaris is correct)

e.g.:
1503c1503
< | Sat May 10 23:59:12 1947 PST
---

| Sat May 10 23:59:12 1947 PDT

Was 1947 PDT or PST ? In eighter case one result is one hour off, Solaris or Linux.

This raises another issue. Why do we distribute expected files with bogus results
in them ? Imho it would be better to only have expected files for rounding issues and
the like. Else the user feels that horology works fine on his machine, but as it looks it only
works on a few.

Andreas

Zeugswetter Andreas SB writes:

Try the attachment with negative values, and tell us whether mktime
returns anything other that -1. Do you have an idea how else we
could determine daylight savings time ?

mktime always returns -1 for tm's that might expect to return a
negative number. In those cases the tm is not normalized and
tm_isdst is set to -1. When mktime returns zero or positive then tm
is normalized and tm_isdst is set to 0 or 1.

localtime sets all the fields of tm correctly, including tm_isdst, for
all values of time_t, including negative ones. When I say correctly,
there is the usual limitation that the rules to specify when DST is in
force cannot express a variation from year to year. (You can specify
e.g. the last Sunday in a month.)

My observations were consistent across AIX 4.1.5, 4.2.1, and 4.3.3.

If you have a time_t, then you can use localtime to determine DST. If
you have a tm then you cannot work out DST for dates before the epoch.
One workaround would be to add 4*n to tm_year and subtract (365*4+1)
*24*60*60*n from the time_t returned. (All leap years are multiples
of 4 in the range 1901 to 2038. If tm_wday is wanted, that will need
to be adjusted as well.) But don't you do time interval arithmetic
using PostgreSQL date types rather than accepting the limitations of
POSIX/UNIX?
--
Pete Forman -./\.- Disclaimer: This post is originated
WesternGeco -./\.- by myself and does not represent
pete.forman@westerngeco.com -./\.- opinion of Schlumberger, Baker
http://www.crosswinds.net/~petef -./\.- Hughes or their divisions.

Pete Forman writes:

One workaround would be to add 4*n to tm_year and subtract (365*4+1)
*24*60*60*n from the time_t returned. (All leap years are multiples
of 4 in the range 1901 to 2038. If tm_wday is wanted, that will need
to be adjusted as well.)

FWIW, that should be to add 28*n to tm_year and subtract (365*4+1)*7
*24*60*60*n from the time_t returned. That calculates tm_wday
correctly.

Also I should have been more explicit that this applies only to AIX
and IRIX. Those return -1 from mktime(year < 1970) and do not allow
DST rules to vary from year to year. Linux and Solaris have more
capable date libraries.
--
Pete Forman http://www.bedford.waii.com/wsdev/petef/PeteF_links.html
WesternGeco http://www.crosswinds.net/~petef
Manton Lane, Bedford, mailto:pete.forman@westerngeco.com
MK41 7PA, UK tel:+44-1234-224798 fax:+44-1234-224804

As I see it, the Linux results are also not 100 % correct in respect to dates
before 1970. (based on assumption that Solaris is correct)
< | Sat May 10 23:59:12 1947 PST

| Sat May 10 23:59:12 1947 PDT

Was 1947 PDT or PST ? In eighter case one result is one hour off, Solaris or Linux.

Yes, I've seen this before. As I mentioned, Solaris does a particularly
good job of including the variations in DST definitions around WWII and
earlier. In the US, there are several different conventions used in
those years, presumably based on a need for energy conservation and
perhaps to maximize production efficiency.

This raises another issue. Why do we distribute expected files with bogus results
in them?

It depends on what you mean by "bogus". imho we should (and do)
distribute "expected" files which reflect the results expected for a
particular machine -- based on a careful analysis of the results from
that machine from an expert, such as you are doing with AIX. Your
results are incremental differences from some other "standard machine",
which has also been carefully analyzed. By definition, the "standard
machine" has been and is a Linux box, for the simple historical reason
that this was my machine at the time that scrappy and I resurrected the
regression tests several years ago. But it is a good choice for the
standard, since that style of machine has a large installed base and the
cost of entry for someone else wanting to participate is very low.

If I understand the alternatives you are considering, the other choice
is to distribute "expected" files which reflect what we think a machine
should produce if it behaved the way we think it should. That doesn't
really help anyone, since a tester would have to derive from first
principles the correctness of the test results each time they run the
test.

Instead, we document the current behavior, and the regression tests can
now catch *changes* in behavior, to be analyzed just as you are doing.
If AIX fixes their mktime() and timezone behavior, you (or someone else
running AIX) will see it, evaluate it, and adjust the regression
"expected" files to reflect this.

- Thomas

FWIW, that should be to add 28*n to tm_year and subtract (365*4+1)*7
*24*60*60*n from the time_t returned. That calculates tm_wday
correctly.
Also I should have been more explicit that this applies only to AIX
and IRIX. Those return -1 from mktime(year < 1970) and do not allow
DST rules to vary from year to year. Linux and Solaris have more
capable date libraries.

Oh, so AIX and IRIX have just one-line time zone databases? Yuck.

How about having some #if BROKEN_TIMEZONE_DATABASE code which uses both
mktime() and localtime() to derive the correct time zone? That is, call
mktime to get a time_t, then call localtime() to get the time zone info,
but only on platforms which do not get a complete result from just the
call to mktime(). In fact, we *could* check for tm->tm_isdst coming back
"-1" for every platform, then call localtime() to make a last stab at
getting a good value.

Comments?

- Thomas

Oh, so AIX and IRIX have just one-line time zone databases? Yuck.

How about having some #if BROKEN_TIMEZONE_DATABASE code which uses both
mktime() and localtime() to derive the correct time zone? That is, call
mktime to get a time_t, then call localtime() to get the time zone info,
but only on platforms which do not get a complete result from just the
call to mktime(). In fact, we *could* check for tm->tm_isdst coming back
"-1" for every platform, then call localtime() to make a last stab at
getting a good value.

How would we construct a valid time_t from the struct tm without mktime?

Andreas

How about having some #if BROKEN_TIMEZONE_DATABASE code which uses both
mktime() and localtime() to derive the correct time zone? That is, call
mktime to get a time_t, then call localtime() to get the time zone info,
but only on platforms which do not get a complete result from just the
call to mktime(). In fact, we *could* check for tm->tm_isdst coming back
"-1" for every platform, then call localtime() to make a last stab at
getting a good value.

How would we construct a valid time_t from the struct tm without mktime?

If I understand the info you have given previously, it should be
possible to get a valid tm->tm_isdst by the following sequence of calls:

// call mktime() which might return a "-1" for DST
time = mktime(tm);
// time is now a correct GMT time
// localtime() *is* allowed to return a good tm->tm_isdst
// even for "negative" time_t values.
// I thought I understood this from Andreas' info...
newtm = localtime(time);
// use the new flag for something useful...
dstflag = newtm->tm_isdst;

Yes?

- Thomas

I have machines running AIX 4.1.5, 4.2.1, and 4.3.3 if you would like
to send me your test programs.

I haven't yet actually fixed the code, but will post patches when I've
done so (assuming that a fix is possible).

- Thomas

How about having some #if BROKEN_TIMEZONE_DATABASE code which uses both
mktime() and localtime() to derive the correct time zone? That is, call
mktime to get a time_t, then call localtime() to get the time zone info,
but only on platforms which do not get a complete result from just the
call to mktime(). In fact, we *could* check for tm->tm_isdst coming back
"-1" for every platform, then call localtime() to make a last stab at
getting a good value.

How would we construct a valid time_t from the struct tm

without mktime?

If I understand the info you have given previously, it should be
possible to get a valid tm->tm_isdst by the following
sequence of calls:

// call mktime() which might return a "-1" for DST
time = mktime(tm);
// time is now a correct GMT time

Unfortunately the returned time is -1 for all dates before 1970
(on AIX and, as I understood, IRIX) :-(
(can you send someone from IBM, that I can shout at, to releive my anger)

Andreas

Thomas Lockhart writes:

I haven't yet actually fixed the code, but will post patches when
I've done so (assuming that a fix is possible).

The normalization in this example program which subtracts 34 years
seems to work OK. I've run it on AIX, IRIX, Linux and Solaris. Some
examples follow.

AIX:
$ TZ=PST8PDT ago34 851995921
Local Mon Dec 30 17:32:01 1996 PST PST
1996-12-30T17:32:01, wday=1, yday=364, isdst = 0
UTC Tue Dec 31 01:32:01 1996 PST PST
1996-12-31T01:32:01, wday=2, yday=365, isdst = 0
Local Sun Dec 30 17:32:01 1962 PST PST
1962-12-30T17:32:01, wday=0, yday=363, isdst = 0
UTC Mon Dec 31 01:32:01 1962 PST PST
1962-12-31T01:32:01, wday=1, yday=364, isdst = 0

Linux:
$ TZ=America/Los_Angeles ago34 426475921
Local Thu Jul 07 18:32:01 1983 PDT -0700
1983-07-07T18:32:01, wday=4, yday=187, isdst = 1
UTC Fri Jul 08 01:32:01 1983 GMT +0000
1983-07-08T01:32:01, wday=5, yday=188, isdst = 0
Local Thu Jul 07 18:32:01 1949 PST -0800
1949-07-07T18:32:01, wday=4, yday=187, isdst = 0
UTC Fri Jul 08 02:32:01 1949 GMT +0000
1949-07-08T02:32:01, wday=5, yday=188, isdst = 0

Here is the program. The call to localtime(&t_ago) is redundant and
hence the adjustment of t_ago can be skipped. It is in this program
as a sanity check.

As it stands, this program assumes that the input and resulting date
are in the usual UNIX range of [1901, 2038]. I presume that there is
code in place that checks the range of dates.

Here is the program. The call to localtime(&t_ago) is redundant and
hence the adjustment of t_ago can be skipped. It is in this program
as a sanity check.
As it stands, this program assumes that the input and resulting date
are in the usual UNIX range of [1901, 2038]. I presume that there is
code in place that checks the range of dates.

Interesting idea. I'm not sure that assuming that timezones from 1943
are the same as timezones from 2013 (they are not, at least in the US)
is any more valid than just accepting the result from your system. I'd
like to explore more possibilities before we settle on a solution.

Perhaps I should just add checks to assume an unspecified time zone wrt
output formatting if the tm_isdst flag comes back as "-1"?

I'll have to look at the ramifications for input times and for
dump/restore operations. Does you system respect the TZ or PGTZ
environment variable?

- Thomas

Thomas Lockhart writes:

Here is the program. The call to localtime(&t_ago) is redundant
and hence the adjustment of t_ago can be skipped. It is in this
program as a sanity check.
As it stands, this program assumes that the input and resulting
date are in the usual UNIX range of [1901, 2038]. I presume that
there is code in place that checks the range of dates.

Interesting idea. I'm not sure that assuming that timezones from
1943 are the same as timezones from 2013 (they are not, at least in
the US) is any more valid than just accepting the result from your
system. I'd like to explore more possibilities before we settle on
a solution.

As far as AIX and IRIX are concerned the timezones _are_ the same. No
variation of rules from year to year is possible. You are not going
to work out DST rules for earlier years without incorporating third
party libraries. As I understand it PostgreSQL undertakes to
calculate dates only as accurately as the underlying OS allows.

The result of mktime for year < 1970 is always -1 and the struct tm is
not normalized; tm_isdst, tm_wday and tm_yday are not calculated. I
can't see that being a useful result.

Perhaps I should just add checks to assume an unspecified time zone
wrt output formatting if the tm_isdst flag comes back as "-1"?

I'm letting the system functions worry about applying time zone and
DST offsets. At no point do I use tm_isdst, except to set it to and
test it for -1.

Thinking about that a bit more, I think that tm_isdst should not be
written into. It would be better to set, say, tm_wday to -1 and
change the test for failure to (t_ago == -1 && local.tm_wday == -1).
tm_isdst should be left as 0 or 1 to help out around the times of
transition to or from DST.

I'll have to look at the ramifications for input times and for
dump/restore operations. Does you system respect the TZ or PGTZ
environment variable?

My code uses localtime and mktime which depend on TZ. There is no
dependency on PGTZ, unless somewhere else in postgres there is an
equivalent of setenv(TZ=getenv(PGTZ)).
--
Pete Forman -./\.- Disclaimer: This post is originated
WesternGeco -./\.- by myself and does not represent
pete.forman@westerngeco.com -./\.- opinion of Schlumberger, Baker
http://www.crosswinds.net/~petef -./\.- Hughes or their divisions.

Pete Forman <pete.forman@westerngeco.com> writes:

Thinking about that a bit more, I think that tm_isdst should not be
written into.

IIRC, setting isdst to -1 was necessary to get the right behavior across
DST boundaries on more-mainstream systems. I do not think it's
acceptable to do worse on systems with good time libraries in order to
improve behavior on fundamentally broken ones.

regards, tom lane

Pete Forman <pete.forman@westerngeco.com> writes:

Thinking about that a bit more, I think that tm_isdst should not be
written into.

IIRC, setting isdst to -1 was necessary to get the right
behavior across
DST boundaries on more-mainstream systems. I do not think it's
acceptable to do worse on systems with good time libraries in order to
improve behavior on fundamentally broken ones.

Yes, the annoyance is, that localtime works for dates before 1970
but mktime doesn't. Best would probably be to assume no DST before
1970 on AIX and IRIX.

Andreas

As far as AIX and IRIX are concerned the timezones _are_ the same. No
variation of rules from year to year is possible. You are not going
to work out DST rules for earlier years without incorporating third
party libraries. As I understand it PostgreSQL undertakes to
calculate dates only as accurately as the underlying OS allows.

Yes. Hence my reluctance to have code which does time-shifting to get
time zones for all platforms. Perhaps it could/should be a configure
test? And then we can have a "HAVE_SIMPLISTIC_TZ" (or whatever) #define
in the code to enable hacks around the problem?

The concern led to my suggestion that we should omit timezone fields
from output -- basically do the equivalent of pre-1901 handling using
GMT -- if DST is not resolved correctly (but I'm still not sure if this
will pan out).

I'll have to look at the ramifications for input times and for
dump/restore operations. Does you system respect the TZ or PGTZ
environment variable?

My code uses localtime and mktime which depend on TZ. There is no
dependency on PGTZ, unless somewhere else in postgres there is an
equivalent of setenv(TZ=getenv(PGTZ)).

Yes there is.

- Thomas

Zeugswetter Andreas SB <ZeugswetterA@wien.spardat.at> writes:

Yes, the annoyance is, that localtime works for dates before 1970
but mktime doesn't. Best would probably be to assume no DST before
1970 on AIX and IRIX.

That seems like a reasonable answer to me, especially since we have
other platforms that behave that way. How can we do this --- just
test for isdst = -1 after the call, and assume that means failure?

regards, tom lane

Yes, the annoyance is, that localtime works for dates before 1970
but mktime doesn't. Best would probably be to assume no DST before
1970 on AIX and IRIX.

That seems like a reasonable answer to me, especially since we have
other platforms that behave that way. How can we do this --- just
test for isdst = -1 after the call, and assume that means failure?

Since we have tests of the form (tm_isdst > 0) ? ... in other parts of the code
this would imho be sufficient.
BTW, I would use the above check for DST in all parts of the code.
Currently we eighter have (tm_isdst ? ....) or the above form.

Andreas

Tom Lane writes:

Pete Forman <pete.forman@westerngeco.com> writes:

Thinking about that a bit more, I think that tm_isdst should not
be written into.

IIRC, setting isdst to -1 was necessary to get the right behavior
across DST boundaries on more-mainstream systems. I do not think
it's acceptable to do worse on systems with good time libraries in
order to improve behavior on fundamentally broken ones.

A footnote in the C89 (and C99) standard says:

Thus, a positive or zero value for tm_isdst causes the
mktime function to presume initially that Daylight Saving
Time, respectively, is or is not in effect for the
specified time. A negative value causes it to attempt to
determine whether Daylight Saving Time is in effect for
the specified time.

So tm_isdst being input as 0 or 1 is not forcing the choice of what it
will be on output. It can be important at the end of DST when local
times repeat and the only way to distinguish them is the setting of
this flag.

That is borne out by my observations.

Setting tm_isdst to -1 before calling mktime can make a difference to
the result when the input and result have different DST flags.

It is fairly arbitrary what the answer to this question is: if six
months is subtracted from a to give b, should a.local.hour =
b.local.hour or should a.utc.hour = b.utc.hour? If you want the
former then set tm_isdst = -1 before calling mktime. I'm out of time
now but I'll try and look for some guidance in the SQL standards.
--
Pete Forman -./\.- Disclaimer: This post is originated
WesternGeco -./\.- by myself and does not represent
pete.forman@westerngeco.com -./\.- opinion of Schlumberger, Baker
http://www.crosswinds.net/~petef -./\.- Hughes or their divisions.

Pete Forman <pete.forman@westerngeco.com> writes:

It is fairly arbitrary what the answer to this question is: if six
months is subtracted from a to give b, should a.local.hour =
b.local.hour or should a.utc.hour = b.utc.hour? If you want the
former then set tm_isdst = -1 before calling mktime.

It's not arbitrary: we want the former. Anything else generates
bug reports from people who expect "9/1/2000 - six months" to
produce "3/1/2000", not sometime late in the evening of 2/29/2000.

regards, tom lane

Yes, the annoyance is, that localtime works for dates before 1970
but mktime doesn't. Best would probably be to assume no DST before
1970 on AIX and IRIX.

That seems like a reasonable answer to me, especially since we have
other platforms that behave that way. How can we do this --- just
test for isdst = -1 after the call, and assume that means failure?

Are you working on this, or can you point me to the parts of the code,
that would need change ?

Andreas

Yes, the annoyance is, that localtime works for dates before 1970
but mktime doesn't. Best would probably be to assume no DST before
1970 on AIX and IRIX.

That seems like a reasonable answer to me, especially since we have
other platforms that behave that way. How can we do this --- just
test for isdst = -1 after the call, and assume that means failure?

Are you working on this, or can you point me to the parts of the code,
that would need change ?

Here is a patch that should make AIX and IRIX more happy.
It changes all checks for tm_isdst to (tm_isdst > 0) and fixes
the expected horology file for AIX.

I just now realized, that the new expected file (while still bogous) is more correct
than the old one.
Thanks to Tom for mentioning that the hour should stay the same when subtracting
days from a timestamp.

Please apply.

Andreas

Yes, the annoyance is, that localtime works for dates before 1970
but mktime doesn't. Best would probably be to assume no DST before
1970 on AIX and IRIX.

That seems like a reasonable answer to me, especially since we have
other platforms that behave that way. How can we do this --- just
test for isdst = -1 after the call, and assume that means failure?

Here is a patch that is incremental to the previous patch and makes AIX
not use DST before 1970. The results are now consistent with other
no-DST-before-1970 platforms.

The down side is, that I did not do a configure test, and did not incooperate
IRIX, since I didn't know what define to check.

The correct thing to do instead of the #if defined (_AIX) would be to use
something like #ifdef NO_NEGATIVE_MKTIME and set that with a configure.
Thomas, are you volunteering ?

Andreas

The correct thing to do instead of the #if defined (_AIX) would be to use
something like #ifdef NO_NEGATIVE_MKTIME and set that with a configure.
Thomas, are you volunteering ?

Actually, I can volunteer to be supportive of your efforts ;) I'm
traveling at the moment, and don't have the original thread(s) which
describe in detail what we need to do for platforms I don't have.

If Peter E. would be willing to do a configure test for this mktime()
problem, then you or I can massage the actual code. Peter, is this
something you could pick up?

I do not have the original thread where Andreas describes the behavior
of mktime() on his machine. Andreas, can you suggest a simple configure
test to be used?

- Thomas

The correct thing to do instead of the #if defined (_AIX) would be to use
something like #ifdef NO_NEGATIVE_MKTIME and set that with a configure.

...Andreas, can you suggest a simple configure
test to be used?

#include <time.h>
int main()
{
struct tm tt, *tm=&tt;
int i = -50000000;
tm = localtime (&i);
i = mktime (tm);
if (i != -50000000) /* on AIX this check could also be (i == -1) */
{
printf("ERROR: mktime(3) does not correctly support datetimes before 1970\n");
return(1);
}
}

On my Linux box, where the test passes, the compiler is happier if "i"
is declared as time_t. Any problem on other platforms if we change this?

- Thomas

Zeugswetter Andreas SB writes:

The down side is, that I did not do a configure test, and did not
incooperate IRIX, since I didn't know what define to check.

The correct thing to do instead of the #if defined (_AIX) would be
to use something like #ifdef NO_NEGATIVE_MKTIME and set that with a
configure.

I agree that configure is the way to go. What if someone has
installed a third party library to provide a better mktime() and
localtime()?

But to answer your question, #if defined (__sgi) is a good test for
IRIX, at least with the native compiler. I can't speak for gcc.
--
Pete Forman -./\.- Disclaimer: This post is originated
WesternGeco -./\.- by myself and does not represent
pete.forman@westerngeco.com -./\.- opinion of Schlumberger, Baker
http://www.crosswinds.net/~petef -./\.- Hughes or their divisions.

Pete Forman writes:

I agree that configure is the way to go. What if someone has
installed a third party library to provide a better mktime() and
localtime()?

Exactly. What if someone has a binary PostgreSQL package installed, then
updates his time library to something supposedly binary compatible and
finds out that PostgreSQL still doesn't use the enhanced capabilities?
Runtime behaviour checks are done at runtime, it's as simple as that.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Exactly. What if someone has a binary PostgreSQL package installed, then
updates his time library to something supposedly binary compatible and
finds out that PostgreSQL still doesn't use the enhanced capabilities?
Runtime behaviour checks are done at runtime, it's as simple as that.

I'm not sure I fully appreciate the distinction here. configure will
check for "behavior" of various kinds, including "behavior assumptions"
in the PostgreSQL code.

So the issue for this case is simply whether it is appropriate to check
for behavior at run time on all platforms, even those known to "never"
exhibit this problematic result, or whether we put in a configure-time
check to save the day for the (two) platforms known to have trouble --
without the other supported platforms taking the hit by having to check
even though the check will never fail.

Andreas, Pete and I were advocating the latter view: that the majority
of platforms which happen to be well behaved should run code optimized
for them, while the bad actors can be supported without "#ifdef __AIX__
|| __IRIX__" in our code, but rather with a more helpful "#ifdef
NO_DST_BEFORE_1970" or whatever.

- Thomas

Thomas Lockhart writes:

I'm not sure I fully appreciate the distinction here. configure will
check for "behavior" of various kinds, including "behavior assumptions"
in the PostgreSQL code.

There are two general categories of platform dependencies: One is
compilation system features, for example presence of certain functions,
libraries, header files, argument types, compiler flags. These sort of
things must be determined before compilation can begin. Autoconf is used
to test for these things.

The other category is run-time behaviour, which is the present case of
testing whether mktime() behaves a certain way when given certain
arguments. Another item that has been thrown around over the years is
whether the system supports Unix domain sockets (essentially a run-time
behaviour check of socket()). You do not need to check these things in
configure; you can just do it when the program runs and adjust
accordingly.

More importantly, you *should* *not* do these tests in configure because
these tests will be unreliable in a cross-compilation situation.
Cross-compilation in this context does not only mean compiling between
completely different platforms, but it includes any setup where the build
system is configured differently from the system you're going to run the
system on, including building on a noexec file system, misconfigured
run-time linkers, different user id, or just a different file system
layout on an otherwise identical platform.

I'm not making these things up. Just yesterday there was a message on
this list where someone ran into this problem and his configure run
misbehaved badly. (PostgreSQL currently violates these rules, but that
does not mean that we should make it harder now to clean it up at some
later date.) Admittedly, these situations sound somewhat exotic, but that
does not mean they do not happen, it may merely mean that PostgreSQL is
not fit to perform in these situations.

So the issue for this case is simply whether it is appropriate to check
for behavior at run time on all platforms, even those known to "never"
exhibit this problematic result,

You can make the run-time check #ifdef one or the other platform. Or you
don't check at all and just postulate the misfeature for a set of
platforms. We have done this in at least two cases: The list of
platforms known not to support Unix domain sockets (HAVE_UNIX_SOCKETS),
and the list of platforms known to have a peculiar bug in the accept()
implementation (happen to be both from SCO). I think this is an
appropriate solution in cases where the affected systems can be classified
easily. But as soon as some versions of some of these systems start
implementing Unix sockets (which is indeed the situation we're currently
facing) or SCO fixes their kernels we're going to have to think harder.

So I would currently suggest that you define

#ifdef AIX || IRIX
# define NO_DST_BEFORE_1970
#endif

but when these systems get fixed you might have to run a simple check once
when the system starts up. This is not really a "hit".

Andreas, Pete and I were advocating the latter view: that the majority
of platforms which happen to be well behaved should run code optimized
for them, while the bad actors can be supported without "#ifdef __AIX__
|| __IRIX__" in our code, but rather with a more helpful "#ifdef
NO_DST_BEFORE_1970" or whatever.

Yeah.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Peter Eisentraut <peter_e@gmx.net> writes:

More importantly, you *should* *not* do these tests in configure because
these tests will be unreliable in a cross-compilation situation.
Cross-compilation in this context does not only mean compiling between
completely different platforms, but it includes any setup where the build
system is configured differently from the system you're going to run the
system on, including building on a noexec file system, misconfigured
run-time linkers, different user id, or just a different file system
layout on an otherwise identical platform.

An approach I've followed in the past is to use three-way logic. If
configuring for a native system, compile and run a program which
provides a yes or no answer. When using cross-configuration, set the
configuration variable to ``don't know'' (or, since this a database
group, NULL).

Pull all code which needs to make this test into a single routine.
Let it check the configuration variable. If the variable is ``don't
know,'' then compile in a static variable, and run the required test
once, at run time, and set the static variable accordingly. Then test
the static variable in all future calls.

This is the approach I used in my UUCP code to look for a bad version
of ftime on some versions of SCO Unix--the ftime result would
sometimes run backward.

Ian
Co-author of GNU Autoconf, Automake, and Libtool

Ian Lance Taylor writes:

An approach I've followed in the past is to use three-way logic. If
configuring for a native system, compile and run a program which
provides a yes or no answer. When using cross-configuration, set the
configuration variable to ``don't know'' (or, since this a database
group, NULL).

This would seem to be the right answer, but unfortunately Autoconf is not
smart enough to detect marginal cross-compilation cases in all situations.
Someone had zlib installed in a location where gcc would find it (compiles
okay) but the run-time linker would not (does not run). This is not
detected when AC_PROG_CC runs, but only later on after you have checked
for the libraries.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Peter Eisentraut <peter_e@gmx.net> writes:

Ian Lance Taylor writes:

An approach I've followed in the past is to use three-way logic. If
configuring for a native system, compile and run a program which
provides a yes or no answer. When using cross-configuration, set the
configuration variable to ``don't know'' (or, since this a database
group, NULL).

This would seem to be the right answer, but unfortunately Autoconf is not
smart enough to detect marginal cross-compilation cases in all situations.
Someone had zlib installed in a location where gcc would find it (compiles
okay) but the run-time linker would not (does not run). This is not
detected when AC_PROG_CC runs, but only later on after you have checked
for the libraries.

Hmmm. I would not describe that as a cross-compilation case at all.
The build machine and the host machine are the same. I would describe
that as a case where the compiler search path and the run time library
search path are not the same.

The autoconf tests don't use any extra libraries, so any discrepancies
of this sort should not matter to them. Autoconf tests whether it can
compile and run a simple program; if it can, it assumes that it is not
in a cross-compilation situation.

Clearly differences between search paths matter, but they are not the
same as cross-compilation.

Ian

Ian Lance Taylor writes:

This would seem to be the right answer, but unfortunately Autoconf is not
smart enough to detect marginal cross-compilation cases in all situations.
Someone had zlib installed in a location where gcc would find it (compiles
okay) but the run-time linker would not (does not run). This is not
detected when AC_PROG_CC runs, but only later on after you have checked
for the libraries.

Hmmm. I would not describe that as a cross-compilation case at all.
The build machine and the host machine are the same.

Only for small values of "same". ;-) Sameness is not defined by the names
being spelled identically or the physical coincidence of the hardware.
There are a million things you can do to a system that supposedly preserve
binary compatibility, such as installing vendor patches or changing a
setting in /etc. But if you run a test program is such a situation you're
testing the wrong system.

I would describe that as a case where the compiler search path and the
run time library search path are not the same.

The assumption is surely that the user would set LD_LIBRARY_PATH or
configure his linker before running the program. But nothing guarantees
that he'll actually set the search path to /usr/local/lib, which is what
gcc was searching in this situation.

Clearly differences between search paths matter, but they are not the
same as cross-compilation.

It's not the same as the classical Canadian Cross, but it's still cross
enough to concern me. ;-)

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Peter Eisentraut <peter_e@gmx.net> writes:

This would seem to be the right answer, but unfortunately Autoconf is not
smart enough to detect marginal cross-compilation cases in all situations.
Someone had zlib installed in a location where gcc would find it (compiles
okay) but the run-time linker would not (does not run). This is not
detected when AC_PROG_CC runs, but only later on after you have checked
for the libraries.

Hmmm. I would not describe that as a cross-compilation case at all.
The build machine and the host machine are the same.

Only for small values of "same". ;-) Sameness is not defined by the names
being spelled identically or the physical coincidence of the hardware.
There are a million things you can do to a system that supposedly preserve
binary compatibility, such as installing vendor patches or changing a
setting in /etc. But if you run a test program is such a situation you're
testing the wrong system.

I believe that terminology is important in technical issues, and in
any case I'm pedantic by nature. So I am going to disagree. When you
build a program on the system on which it is going to run, that is not
cross-compilation. I agree that if you build a program on one system,
and then copy it (either physically or via something like NFS) to
another system, and run it on that other system, then that may be
cross-compilation.

It may be that the program in this case was being moved from one
machine to another, and I missed it. However, unless that was the
case, a difference in search path between the compiler and the runtime
linker is not a cross-compilation issue. A difference in where a
library appears from one machine to another is a cross-compilation
issue.

I believe that it is important to describe problems correctly in order
to understand how to fix them correctly. If the problem is ``autoconf
does not correctly diagnose a case of cross-compilation,'' that
suggests one sort of fix. If the problem is ``the compiler and the
runtime linker have different search paths,'' that suggests a
different sort of fix.

To return to the issue which started this thread, I believe it was an
issue concerning the behaviour of the mktime library call. The
behaviour of that library call is not going to be affected by the
location of a shared library. The approach I suggested should suffice
to address the particular problem at hand. Do you disagree?

The issue in which the compiler and the runtime linker use different
search paths is fixed by either changing the compiler search path, to
by default only choose shared libraries from directories which the
runtime linker will search, or by changing the library installation
procedure, either put the library in /usr/lib, or on systems which
support ld.so.conf arrange to put the directory in ld.so.conf.

Note that if you link with -lz, at least some versions of the GNU
linker will issue a warning that the library will not be found by the
runtime linker.

Ian

Peter Eisentraut writes:

What if someone has a binary PostgreSQL package installed, then
updates his time library to something supposedly binary compatible
and finds out that PostgreSQL still doesn't use the enhanced
capabilities?

You are too generous. If someone downloads a binary package it should
not be expected to be able to take advantage of non standard features
of the platform. It is reasonable that they should compile it from
source to get the most from it.
--
Pete Forman -./\.- Disclaimer: This post is originated
WesternGeco -./\.- by myself and does not represent
pete.forman@westerngeco.com -./\.- opinion of Schlumberger, Baker
http://www.crosswinds.net/~petef -./\.- Hughes or their divisions.