gitmaster server problem?

Started by Bruce Momjianover 1 year ago29 messages
Jump to latest
#1Bruce Momjian
bruce@momjian.us

About 20 minutes ago I starting getting gitmaster pull errors:

$ git pull ssh://git@gitmaster.postgresql.org/postgresql.git
ssh: connect to host gitmaster.postgresql.org port 22: Connection timed
out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Public git seems fine:

$ git pull https://git.postgresql.org/git/postgresql.git
From https://git.postgresql.org/git/postgresql
* branch HEAD -> FETCH_HEAD
Already up to date.

Can anyone confirm or report on the cause?

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Bruce Momjian (#1)
Re: gitmaster server problem?

Bruce Momjian <bruce@momjian.us> writes:

About 20 minutes ago I starting getting gitmaster pull errors:
...
Can anyone confirm or report on the cause?

Yeah, for me it's hanging up at the connection establishment
stage ("ssh -fMN gitmaster.postgresql.org"). Traceroute gets
as far as some rackspace.net machines, but gemulon itself
isn't answering.

regards, tom lane

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Lane (#2)
Re: gitmaster server problem?

I wrote:

Bruce Momjian <bruce@momjian.us> writes:

About 20 minutes ago I starting getting gitmaster pull errors:
Can anyone confirm or report on the cause?

Yeah, for me it's hanging up at the connection establishment
stage ("ssh -fMN gitmaster.postgresql.org"). Traceroute gets
as far as some rackspace.net machines, but gemulon itself
isn't answering.

ssh and git pull are working again now for me.

Oddly, "ping gemulon.postgresql.org" is still showing 98%+
packet drop rate. Not sure if that's normal.

regards, tom lane

#4Matthias van de Meent
boekewurm+postgres@gmail.com
In reply to: Bruce Momjian (#1)
Re: gitmaster server problem?

On Sat, 17 Aug 2024 at 18:41, Bruce Momjian <bruce@momjian.us> wrote:

About 20 minutes ago I starting getting gitmaster pull errors:

$ git pull ssh://git@gitmaster.postgresql.org/postgresql.git
ssh: connect to host gitmaster.postgresql.org port 22: Connection timed
out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[...]

Can anyone confirm or report on the cause?

Additional infra that doesn't seem to work right now: mailing list
archives [0]https://www.postgresql.org/list/pgsql-hackers/2024-08/ seem to fail at a 503 produced by Varnish Cache server:
Error 503 Backend fetch failed. Maybe more of infra is down, or
otherwise under maintenance?

Kind regards,

Matthias van de Meent

[0]: https://www.postgresql.org/list/pgsql-hackers/2024-08/

#5Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matthias van de Meent (#4)
Re: gitmaster server problem?

Matthias van de Meent <boekewurm+postgres@gmail.com> writes:

Additional infra that doesn't seem to work right now: mailing list
archives [0] seem to fail at a 503 produced by Varnish Cache server:
Error 503 Backend fetch failed. Maybe more of infra is down, or
otherwise under maintenance?

Yeah, I'm seeing that too; not at the top level of the site but
when you try to drill down to individual messages.

commitfest.postgresql.org isn't responding either,
nor wiki.postgresql.org.

regards, tom lane

#6Bruce Momjian
bruce@momjian.us
In reply to: Tom Lane (#5)
Re: gitmaster server problem?

On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote:

Matthias van de Meent <boekewurm+postgres@gmail.com> writes:

Additional infra that doesn't seem to work right now: mailing list
archives [0] seem to fail at a 503 produced by Varnish Cache server:
Error 503 Backend fetch failed. Maybe more of infra is down, or
otherwise under maintenance?

Yeah, I'm seeing that too; not at the top level of the site but
when you try to drill down to individual messages.

commitfest.postgresql.org isn't responding either,
nor wiki.postgresql.org.

Okay, thanks for the confirmation. I emailed pginfra at the same time I
posted the first email, so they are aware. I have not received a reply
from them yet.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.

#7Adrian Klaver
adrian.klaver@gmail.com
In reply to: Tom Lane (#3)
Re: gitmaster server problem?

On 8/17/24 10:22, Tom Lane wrote:

I wrote:

Bruce Momjian <bruce@momjian.us> writes:

About 20 minutes ago I starting getting gitmaster pull errors:
Can anyone confirm or report on the cause?

Yeah, for me it's hanging up at the connection establishment
stage ("ssh -fMN gitmaster.postgresql.org"). Traceroute gets
as far as some rackspace.net machines, but gemulon itself
isn't answering.

ssh and git pull are working again now for me.

Oddly, "ping gemulon.postgresql.org" is still showing 98%+
packet drop rate. Not sure if that's normal.

I don't even get that, it is still dying at:

9 cored-aggr181a-1.dfw3.rackspace.net (148.62.65.5)

regards, tom lane

--
Adrian Klaver
adrian.klaver@aklaver.com

#8Bruce Momjian
bruce@momjian.us
In reply to: Tom Lane (#3)
Re: gitmaster server problem?

On Sat, Aug 17, 2024 at 01:22:18PM -0400, Tom Lane wrote:

I wrote:

Bruce Momjian <bruce@momjian.us> writes:

About 20 minutes ago I starting getting gitmaster pull errors:
Can anyone confirm or report on the cause?

Yeah, for me it's hanging up at the connection establishment
stage ("ssh -fMN gitmaster.postgresql.org"). Traceroute gets
as far as some rackspace.net machines, but gemulon itself
isn't answering.

ssh and git pull are working again now for me.

ssh and git pull are still failing for me to gitmaster.postgresql.org.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.

#9Bruce Momjian
bruce@momjian.us
In reply to: Bruce Momjian (#8)
Re: gitmaster server problem?

On Sat, Aug 17, 2024 at 02:24:54PM -0400, Bruce Momjian wrote:

On Sat, Aug 17, 2024 at 01:22:18PM -0400, Tom Lane wrote:

I wrote:

Bruce Momjian <bruce@momjian.us> writes:

About 20 minutes ago I starting getting gitmaster pull errors:
Can anyone confirm or report on the cause?

Yeah, for me it's hanging up at the connection establishment
stage ("ssh -fMN gitmaster.postgresql.org"). Traceroute gets
as far as some rackspace.net machines, but gemulon itself
isn't answering.

ssh and git pull are working again now for me.

ssh and git pull are still failing for me to gitmaster.postgresql.org.

Joe Conway, CC'ed, is aware of the issue and is now working on it.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.

#10Bruce Momjian
bruce@momjian.us
In reply to: Bruce Momjian (#9)
Re: gitmaster server problem?

On Sat, Aug 17, 2024 at 02:53:44PM -0400, Bruce Momjian wrote:

On Sat, Aug 17, 2024 at 02:24:54PM -0400, Bruce Momjian wrote:

On Sat, Aug 17, 2024 at 01:22:18PM -0400, Tom Lane wrote:

I wrote:

Bruce Momjian <bruce@momjian.us> writes:

About 20 minutes ago I starting getting gitmaster pull errors:
Can anyone confirm or report on the cause?

Yeah, for me it's hanging up at the connection establishment
stage ("ssh -fMN gitmaster.postgresql.org"). Traceroute gets
as far as some rackspace.net machines, but gemulon itself
isn't answering.

ssh and git pull are working again now for me.

ssh and git pull are still failing for me to gitmaster.postgresql.org.

Joe Conway, CC'ed, is aware of the issue and is now working on it.

Joe reports that everything should be working normally now.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.

#11Tom Lane
tgl@sss.pgh.pa.us
In reply to: Bruce Momjian (#10)
Re: gitmaster server problem?

Bruce Momjian <bruce@momjian.us> writes:

Joe reports that everything should be working normally now.

ACK - things look good from here. (I did have to restart my
ssh tunnel to gitmaster again.)

Thanks Joe!

regards, tom lane

#12Joe Conway
mail@joeconway.com
In reply to: Matthias van de Meent (#4)
Re: gitmaster server problem?

On 8/17/24 13:59, Matthias van de Meent wrote:

On Sat, 17 Aug 2024 at 18:41, Bruce Momjian <bruce@momjian.us> wrote:

About 20 minutes ago I starting getting gitmaster pull errors:

$ git pull ssh://git@gitmaster.postgresql.org/postgresql.git
ssh: connect to host gitmaster.postgresql.org port 22: Connection timed
out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[...]

Can anyone confirm or report on the cause?

Additional infra that doesn't seem to work right now: mailing list
archives [0] seem to fail at a 503 produced by Varnish Cache server:
Error 503 Backend fetch failed. Maybe more of infra is down, or
otherwise under maintenance?

Kind regards,

Matthias van de Meent

[0] https://www.postgresql.org/list/pgsql-hackers/2024-08/

Just to close this out -- problem found and fixed about an hour ago.
Sorry for the interruption.

--
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

#13Jelte Fennema-Nio
postgres@jeltef.nl
In reply to: Joe Conway (#12)
Re: gitmaster server problem?

On Sat, 17 Aug 2024 at 22:05, Joe Conway <mail@joeconway.com> wrote:

Just to close this out -- problem found and fixed about an hour ago.
Sorry for the interruption.

Whatever the problem was it seems to have returned

#14Joe Conway
mail@joeconway.com
In reply to: Jelte Fennema-Nio (#13)
Re: gitmaster server problem?

On 8/19/24 05:04, Jelte Fennema-Nio wrote:

On Sat, 17 Aug 2024 at 22:05, Joe Conway <mail@joeconway.com> wrote:

Just to close this out -- problem found and fixed about an hour ago.
Sorry for the interruption.

Whatever the problem was it seems to have returned

What specifically? I am not seeing anything at the moment.

--
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

#15Jelte Fennema-Nio
postgres@jeltef.nl
In reply to: Joe Conway (#14)
Re: gitmaster server problem?

On Mon, 19 Aug 2024 at 13:39, Joe Conway <mail@joeconway.com> wrote:

What specifically? I am not seeing anything at the moment.

It seems to be working again fine. But before
https://git.postgresql.org was throwing 502 and 503 errors. Depending
on the request either by Varnish or nginx.

#16Robert Haas
robertmhaas@gmail.com
In reply to: Jelte Fennema-Nio (#15)
Re: gitmaster server problem?

On Mon, Aug 19, 2024 at 7:57 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

On Mon, 19 Aug 2024 at 13:39, Joe Conway <mail@joeconway.com> wrote:

What specifically? I am not seeing anything at the moment.

It seems to be working again fine. But before
https://git.postgresql.org was throwing 502 and 503 errors. Depending
on the request either by Varnish or nginx.

I'm still unable to access any https://git.postgresql.org/ URL.

--
Robert Haas
EDB: http://www.enterprisedb.com

#17Jelte Fennema-Nio
postgres@jeltef.nl
In reply to: Robert Haas (#16)
Re: gitmaster server problem?

On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote:

I'm still unable to access any https://git.postgresql.org/ URL.

Yeah it's broken for me again too.

#18Jelte Fennema-Nio
postgres@jeltef.nl
In reply to: Jelte Fennema-Nio (#17)
Re: gitmaster server problem?

On Mon, 19 Aug 2024 at 14:05, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote:

I'm still unable to access any https://git.postgresql.org/ URL.

Yeah it's broken for me again too.

In case the actual error is helpful (although I doubt it):

Error 503 Backend fetch failed

Backend fetch failed

Guru Meditation:

XID: 1030914118

________________________________

Varnish cache server

#19Joe Conway
mail@joeconway.com
In reply to: Jelte Fennema-Nio (#18)
Re: gitmaster server problem?

On 8/19/24 08:06, Jelte Fennema-Nio wrote:

On Mon, 19 Aug 2024 at 14:05, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote:

I'm still unable to access any https://git.postgresql.org/ URL.

Yeah it's broken for me again too.

In case the actual error is helpful (although I doubt it):

Error 503 Backend fetch failed

Backend fetch failed

Guru Meditation:

XID: 1030914118

I am not seeing any errors here. nagios monitoring is showing an ipv6
error though. Not the same issue as yesterday.

--
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

#20jian he
jian.universality@gmail.com
In reply to: Bruce Momjian (#6)
Re: gitmaster server problem?

On Mon, Aug 19, 2024 at 9:56 PM Bruce Momjian <bruce@momjian.us> wrote:

as of now.
cgit working
https://git.postgresql.org/cgit/postgresql.git/

git not working
( 403 Forbidden nginx)
https://git.postgresql.org/git/postgresql.git/

#21Dave Page
dpage@pgadmin.org
In reply to: Bruce Momjian (#6)
#22Magnus Hagander
magnus@hagander.net
In reply to: Tom Lane (#5)
#23Bruce Momjian
bruce@momjian.us
In reply to: Joe Conway (#19)
#24Tom Lane
tgl@sss.pgh.pa.us
In reply to: Magnus Hagander (#22)
#25Dave Page
dpage@pgadmin.org
In reply to: jian he (#20)
#26Dave Page
dpage@pgadmin.org
In reply to: Tom Lane (#24)
#27Magnus Hagander
magnus@hagander.net
In reply to: Dave Page (#21)
#28Adrian Klaver
adrian.klaver@gmail.com
In reply to: Tom Lane (#24)
#29Tom Lane
tgl@sss.pgh.pa.us
In reply to: Magnus Hagander (#27)