Strangely, the same thing does not happen when I do timenow() instead of time(). This is very counter-intuitive, if this is the way it is supposed to work, at least docs should be saying that.
Also, I checked, and its probably not the fmgr cache, since now() is set to be noncacheable...
