server closed the connection unexpectedly

Started by Clodoaldo Pinto Netoover 21 years ago7 messagesgeneral
Jump to latest
#1Clodoaldo Pinto Neto
clodoaldo_pinto@yahoo.com.br

Hi all,

Errors running a psql batch that has worked for weeks without a
problem:
$ uname -a
Linux s1 2.6.6-1.435.2.3 #1 Thu Jul 1 08:25:29 EDT 2004 i686 athlon
i386 GNU/Linux
$ rpm -q postgresql
postgresql-7.4.2-1
$ psql KakaoStats
Bem-vindo ao psql 7.4.2, o terminal iterativo do PostgreSQL.

select insert_usuarios_producao();
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: server closed the
connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: connection to server
was lost

select deleta_antigos()
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

vacuum KakaoStats
psql: FATAL: the database system is starting up

These commands are some of the last of a script which runs normaly
until then.

Regards
Clodoaldo Pinto

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv�rus gr�tis!
http://br.info.mail.yahoo.com/

#2mike g
mike@thegodshalls.com
In reply to: Clodoaldo Pinto Neto (#1)
Re: server closed the connection unexpectedly

It seems that postgresql crashed recently, tried to restart
automatically, and has not succesfully restarted.

You will need to stop the postmaster daemon manually, probably delete
the postmaster.pid file, and restart the postmaster daemon.

Chances are that anyone trying to query the database is not able to....

HTH

Show quoted text

On Fri, 2004-07-16 at 21:05, Clodoaldo Pinto Neto wrote:

Hi all,

Errors running a psql batch that has worked for weeks without a
problem:
$ uname -a
Linux s1 2.6.6-1.435.2.3 #1 Thu Jul 1 08:25:29 EDT 2004 i686 athlon
i386 GNU/Linux
$ rpm -q postgresql
postgresql-7.4.2-1
$ psql KakaoStats
Bem-vindo ao psql 7.4.2, o terminal iterativo do PostgreSQL.

select insert_usuarios_producao();
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: server closed the
connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: connection to server
was lost

select deleta_antigos()
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

vacuum KakaoStats
psql: FATAL: the database system is starting up

These commands are some of the last of a script which runs normaly
until then.

Regards
Clodoaldo Pinto

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv��rus gr��tis!
http://br.info.mail.yahoo.com/

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

#3Clodoaldo Pinto Neto
clodoaldo_pinto@yahoo.com.br
In reply to: mike g (#2)
Re: server closed the connection unexpectedly

There was no crash that I know of.

# su postgres
$ cd /var/lib/pgsql/data
$ cat postmaster.pid
14877
/var/lib/pgsql/data
5432001 393216
$ pg_ctl stop -D ./
waiting for postmaster to shut down......done
$ rm postmaster.pid
rm: impos�vel remover `postmaster.pid': Arquivo ou diret�rio n�o
encontrado #(means file not found)
$ postmaster -D /var/lib/pgsql/data > logfile 2>&1 &
[1]: 19256

Now as user cpn:
$ pg_dump -Fc KakaoStats > KakaoStats.dump
pg_dump: conector (socket) n�o est� aberto #(means not open)
pg_dump: Comando SQL para copiar o conte�do da tabela "usuarios"
falhou: PQendcopy() falhou. #(means failed)
pg_dump: Mensagem de erro do servidor: conector (socket) n�o est�
aberto #(not open)
pg_dump: O comando foi: COPY public.usuarios (usuario, data, n_time,
wus, pontos) TO stdout;

After reboot this was the log file: (forgot to get it before)

# cat logfile
LOG: database system was shut down at 2004-07-17 05:41:04 BRT
LOG: checkpoint record is at 7/C63CFA44
LOG: redo record is at 7/C63CFA44; undo record is at 0/0; shutdown
TRUE
LOG: next transaction ID: 7704; next OID: 426301
LOG: database system is ready
LOG: server process (PID 19261) was terminated by signal 11
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted at 2004-07-17 05:45:22 BRT
LOG: checkpoint record is at 7/C63CFA44
LOG: redo record is at 7/C63CFA44; undo record is at 0/0; shutdown
TRUE
LOG: next transaction ID: 7704; next OID: 426301
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: record with zero length at 7/C63CFA84
LOG: redo is not required
LOG: database system is ready
LOG: received fast shutdown request
LOG: shutting down
LOG: database system is shut down

After reboot it worked OK.
CPN

--- mike g <mike@thegodshalls.com> escreveu: > It seems that postgresql
crashed recently, tried to restart

automatically, and has not succesfully restarted.

You will need to stop the postmaster daemon manually, probably delete
the postmaster.pid file, and restart the postmaster daemon.

Chances are that anyone trying to query the database is not able
to....

HTH

On Fri, 2004-07-16 at 21:05, Clodoaldo Pinto Neto wrote:

Hi all,

Errors running a psql batch that has worked for weeks without a
problem:
$ uname -a
Linux s1 2.6.6-1.435.2.3 #1 Thu Jul 1 08:25:29 EDT 2004 i686 athlon
i386 GNU/Linux
$ rpm -q postgresql
postgresql-7.4.2-1
$ psql KakaoStats
Bem-vindo ao psql 7.4.2, o terminal iterativo do PostgreSQL.

select insert_usuarios_producao();
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: server closed the
connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: connection to

server

was lost

select deleta_antigos()
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

vacuum KakaoStats
psql: FATAL: the database system is starting up

These commands are some of the last of a script which runs normaly
until then.

Regards
Clodoaldo Pinto

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antivírus grátis!
http://br.info.mail.yahoo.com/

---------------------------(end of

broadcast)---------------------------

TIP 6: Have you searched our list archives?

http://archives.postgresql.org

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv�rus gr�tis!
http://br.info.mail.yahoo.com/

#4mike g
mike@thegodshalls.com
In reply to: Clodoaldo Pinto Neto (#3)
Re: server closed the connection unexpectedly

Glad to hear it worked out.

This section indicates it had crashed at some point:
LOG: database system was not properly shut down; automatic recovery in

progress

Mike

Show quoted text

On Sat, 2004-07-17 at 04:30, Clodoaldo Pinto Neto wrote:

There was no crash that I know of.

# su postgres
$ cd /var/lib/pgsql/data
$ cat postmaster.pid
14877
/var/lib/pgsql/data
5432001 393216
$ pg_ctl stop -D ./
waiting for postmaster to shut down......done
$ rm postmaster.pid
rm: impos��vel remover `postmaster.pid': Arquivo ou diret��rio n��o
encontrado #(means file not found)
$ postmaster -D /var/lib/pgsql/data > logfile 2>&1 &
[1] 19256

Now as user cpn:
$ pg_dump -Fc KakaoStats > KakaoStats.dump
pg_dump: conector (socket) n��o est�� aberto #(means not open)
pg_dump: Comando SQL para copiar o conte��do da tabela "usuarios"
falhou: PQendcopy() falhou. #(means failed)
pg_dump: Mensagem de erro do servidor: conector (socket) n��o est��
aberto #(not open)
pg_dump: O comando foi: COPY public.usuarios (usuario, data, n_time,
wus, pontos) TO stdout;

After reboot this was the log file: (forgot to get it before)

# cat logfile
LOG: database system was shut down at 2004-07-17 05:41:04 BRT
LOG: checkpoint record is at 7/C63CFA44
LOG: redo record is at 7/C63CFA44; undo record is at 0/0; shutdown
TRUE
LOG: next transaction ID: 7704; next OID: 426301
LOG: database system is ready
LOG: server process (PID 19261) was terminated by signal 11
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted at 2004-07-17 05:45:22 BRT
LOG: checkpoint record is at 7/C63CFA44
LOG: redo record is at 7/C63CFA44; undo record is at 0/0; shutdown
TRUE
LOG: next transaction ID: 7704; next OID: 426301
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: record with zero length at 7/C63CFA84
LOG: redo is not required
LOG: database system is ready
LOG: received fast shutdown request
LOG: shutting down
LOG: database system is shut down

After reboot it worked OK.
CPN

--- mike g <mike@thegodshalls.com> escreveu: > It seems that postgresql
crashed recently, tried to restart

automatically, and has not succesfully restarted.

You will need to stop the postmaster daemon manually, probably delete
the postmaster.pid file, and restart the postmaster daemon.

Chances are that anyone trying to query the database is not able
to....

HTH

On Fri, 2004-07-16 at 21:05, Clodoaldo Pinto Neto wrote:

Hi all,

Errors running a psql batch that has worked for weeks without a
problem:
$ uname -a
Linux s1 2.6.6-1.435.2.3 #1 Thu Jul 1 08:25:29 EDT 2004 i686 athlon
i386 GNU/Linux
$ rpm -q postgresql
postgresql-7.4.2-1
$ psql KakaoStats
Bem-vindo ao psql 7.4.2, o terminal iterativo do PostgreSQL.

select insert_usuarios_producao();
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: server closed the
connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: connection to

server

was lost

select deleta_antigos()
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

vacuum KakaoStats
psql: FATAL: the database system is starting up

These commands are some of the last of a script which runs normaly
until then.

Regards
Clodoaldo Pinto

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv����rus gr����tis!
http://br.info.mail.yahoo.com/

---------------------------(end of

broadcast)---------------------------

TIP 6: Have you searched our list archives?

http://archives.postgresql.org

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv��rus gr��tis!
http://br.info.mail.yahoo.com/

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

#5Clodoaldo Pinto Neto
clodoaldo_pinto@yahoo.com.br
In reply to: mike g (#4)
Re: server closed the connection unexpectedly

Mike,

Thank you for you reply. There is no sign in /var/log/messages of
system crash or reboot when or before the problem happened. Also I
always stop postmaster with pg_ctl stop and there was sometime that I
last stoped it.

In case it is worth something this is the /var/log/messages from when
the problem was happening:

Jul 17 05:34:07 s1 kernel: <3>Debug: sleeping function called from
invalid context at include/linux/rwsem.h:43
Jul 17 05:34:07 s1 kernel: in_atomic():0, irqs_disabled():1
Jul 17 05:34:07 s1 kernel: [<02116834>] __might_sleep+0x80/0x8a
Jul 17 05:34:07 s1 kernel: [<0213fce7>] rw_vm+0x9b/0x218
Jul 17 05:34:07 s1 kernel: [<0212bf93>] find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: [<0212bf93>] find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: [<02140096>] get_user_size+0x2e/0x55
Jul 17 05:34:07 s1 kernel: [<0212bf93>] find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: [<0211430e>] __is_prefetch+0x1a7/0x295
Jul 17 05:34:07 s1 kernel: [<021156ed>] recalc_task_prio+0x128/0x133
Jul 17 05:34:07 s1 kernel: [<0212bf93>] find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: [<0212bf93>] find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: [<0211466a>] do_page_fault+0x26e/0x446
Jul 17 05:34:07 s1 kernel: [<02115e8c>] __wake_up_common+0x32/0x54
Jul 17 05:34:07 s1 kernel: [<02115ebf>] __wake_up+0x11/0x1a
Jul 17 05:34:07 s1 kernel: [<0213639b>] follow_page_pfn+0xec/0xfd
Jul 17 05:34:07 s1 kernel: [<0213fe44>] rw_vm+0x1f8/0x218
Jul 17 05:34:07 s1 kernel: [<021143fc>] do_page_fault+0x0/0x446
Jul 17 05:34:07 s1 kernel: [<0212bf93>] find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: [<0212c24c>]
do_generic_mapping_read+0xb0/0x2a9
Jul 17 05:34:07 s1 kernel: [<0212c69d>]
__generic_file_aio_read+0x157/0x171
Jul 17 05:34:07 s1 kernel: [<0212c445>] file_read_actor+0x0/0x101
Jul 17 05:34:07 s1 kernel: [<0212c6f7>]
generic_file_aio_read+0x40/0x47
Jul 17 05:34:07 s1 kernel: [<02141f2e>] do_sync_read+0x68/0x9d
Jul 17 05:34:07 s1 kernel: [<0213639b>] follow_page_pfn+0xec/0xfd
Jul 17 05:34:07 s1 kernel: [<021156ed>] recalc_task_prio+0x128/0x133
Jul 17 05:34:07 s1 kernel: [<0214201b>] vfs_read+0xb8/0xe4
Jul 17 05:34:07 s1 kernel: [<021421f4>] sys_read+0x2c/0x42
Jul 17 05:34:07 s1 kernel:
Jul 17 05:34:07 s1 kernel: Unable to handle kernel paging request at
virtual address 00040000
Jul 17 05:34:07 s1 kernel: printing eip:
Jul 17 05:34:07 s1 kernel: 0212bf93
Jul 17 05:34:07 s1 kernel: *pde = 00000000
Jul 17 05:34:07 s1 kernel: Oops: 0000 [#12]
Jul 17 05:34:07 s1 kernel: Modules linked in: nfsd exportfs lockd ipv6
parport_pc lp parport autofs4 sunrpc via_rhine mii sg scsi_mod dm_mod
uhci_hcd ehci_hcd button battery asus_acpi ac ext3 jbd
Jul 17 05:34:07 s1 kernel: CPU: 0
Jul 17 05:34:07 s1 kernel: EIP: 0060:[<0212bf93>] Not tainted
Jul 17 05:34:07 s1 kernel: EFLAGS: 00010006 (2.6.6-1.435.2.3)
Jul 17 05:34:07 s1 kernel: EIP is at find_get_page+0x11/0x24
Jul 17 05:34:07 s1 kernel: eax: 00040000 ebx: 00000000 ecx:
00040000 edx: 00040000
Jul 17 05:34:07 s1 kernel: esi: 00001000 edi: 00001000 ebp:
00000000 esp: 1c206e18
Jul 17 05:34:07 s1 kernel: ds: 007b es: 007b ss: 0068
Jul 17 05:34:07 s1 kernel: Process postmaster (pid: 19187,
threadinfo=1c206000 task=1bcf18b0)
Jul 17 05:34:07 s1 kernel: Stack: 0212c24c 00000000 000094af 247d0e8c
14c8e9c0 14c8ea08 247d0f24 000094a8
Jul 17 05:34:07 s1 kernel: 00000020 00000020 000094af 000094c8
00000020 00000041 00000010 00000020
Jul 17 05:34:07 s1 kernel: 00000000 00000000 1c206eb8 1c206ec0
00000000 00000000 00002000 0212c69d
Jul 17 05:34:07 s1 kernel: Call Trace:
Jul 17 05:34:07 s1 kernel: [<0212c24c>]
do_generic_mapping_read+0xb0/0x2a9
Jul 17 05:34:07 s1 kernel: [<0212c69d>]
__generic_file_aio_read+0x157/0x171
Jul 17 05:34:07 s1 kernel: [<0212c445>] file_read_actor+0x0/0x101
Jul 17 05:34:07 s1 kernel: [<0212c6f7>]
generic_file_aio_read+0x40/0x47
Jul 17 05:34:07 s1 kernel: [<02141f2e>] do_sync_read+0x68/0x9d
Jul 17 05:34:07 s1 kernel: [<0213639b>] follow_page_pfn+0xec/0xfd
Jul 17 05:34:07 s1 kernel: [<021156ed>] recalc_task_prio+0x128/0x133
Jul 17 05:34:07 s1 kernel: [<0214201b>] vfs_read+0xb8/0xe4
Jul 17 05:34:07 s1 kernel: [<021421f4>] sys_read+0x2c/0x42
Jul 17 05:34:07 s1 kernel:
Jul 17 05:34:07 s1 kernel: Code: 8b 00 a9 00 00 08 00 74 03 8b 51 0c ff
42 04 fb 89 c8 c3 fa

Regards,
CPN

--- mike g <mike@thegodshalls.com> escreveu: > Glad to hear it worked
out.

This section indicates it had crashed at some point:
LOG: database system was not properly shut down; automatic recovery
in

progress

Mike

On Sat, 2004-07-17 at 04:30, Clodoaldo Pinto Neto wrote:

There was no crash that I know of.

# su postgres
$ cd /var/lib/pgsql/data
$ cat postmaster.pid
14877
/var/lib/pgsql/data
5432001 393216
$ pg_ctl stop -D ./
waiting for postmaster to shut down......done
$ rm postmaster.pid
rm: imposível remover `postmaster.pid': Arquivo ou diretório não
encontrado #(means file not found)
$ postmaster -D /var/lib/pgsql/data > logfile 2>&1 &
[1] 19256

Now as user cpn:
$ pg_dump -Fc KakaoStats > KakaoStats.dump
pg_dump: conector (socket) não está aberto #(means not open)
pg_dump: Comando SQL para copiar o conteúdo da tabela "usuarios"
falhou: PQendcopy() falhou. #(means failed)
pg_dump: Mensagem de erro do servidor: conector (socket) não está
aberto #(not open)
pg_dump: O comando foi: COPY public.usuarios (usuario, data,

n_time,

wus, pontos) TO stdout;

After reboot this was the log file: (forgot to get it before)

# cat logfile
LOG: database system was shut down at 2004-07-17 05:41:04 BRT
LOG: checkpoint record is at 7/C63CFA44
LOG: redo record is at 7/C63CFA44; undo record is at 0/0; shutdown
TRUE
LOG: next transaction ID: 7704; next OID: 426301
LOG: database system is ready
LOG: server process (PID 19261) was terminated by signal 11
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted at 2004-07-17 05:45:22 BRT
LOG: checkpoint record is at 7/C63CFA44
LOG: redo record is at 7/C63CFA44; undo record is at 0/0; shutdown
TRUE
LOG: next transaction ID: 7704; next OID: 426301
LOG: database system was not properly shut down; automatic

recovery in

progress
LOG: record with zero length at 7/C63CFA84
LOG: redo is not required
LOG: database system is ready
LOG: received fast shutdown request
LOG: shutting down
LOG: database system is shut down

After reboot it worked OK.
CPN

--- mike g <mike@thegodshalls.com> escreveu: > It seems that

postgresql

crashed recently, tried to restart

automatically, and has not succesfully restarted.

You will need to stop the postmaster daemon manually, probably

delete

the postmaster.pid file, and restart the postmaster daemon.

Chances are that anyone trying to query the database is not able
to....

HTH

On Fri, 2004-07-16 at 21:05, Clodoaldo Pinto Neto wrote:

Hi all,

Errors running a psql batch that has worked for weeks without a
problem:
$ uname -a
Linux s1 2.6.6-1.435.2.3 #1 Thu Jul 1 08:25:29 EDT 2004 i686

athlon

i386 GNU/Linux
$ rpm -q postgresql
postgresql-7.4.2-1
$ psql KakaoStats
Bem-vindo ao psql 7.4.2, o terminal iterativo do PostgreSQL.

select insert_usuarios_producao();
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: server closed

the

connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
psql:/KakaoStats/scripts/sql/kakaoStats.sql:92: connection to

server

was lost

select deleta_antigos()
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

vacuum KakaoStats
psql: FATAL: the database system is starting up

These commands are some of the last of a script which runs

normaly

until then.

Regards
Clodoaldo Pinto

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv�rus gr�tis!
http://br.info.mail.yahoo.com/

#6Alvaro Herrera
alvherre@dcc.uchile.cl
In reply to: Clodoaldo Pinto Neto (#5)
Re: server closed the connection unexpectedly

On Sun, Jul 18, 2004 at 08:18:07AM -0300, Clodoaldo Pinto Neto wrote:

Thank you for you reply. There is no sign in /var/log/messages of
system crash or reboot when or before the problem happened. Also I
always stop postmaster with pg_ctl stop and there was sometime that I
last stoped it.

This is clearly a kernel problem. Postgres does not have enough
privileges to trigger this kind of mess.

Or maybe you have bad hardware; RAM, most likely. Have you run hardware
testing tools, like Memtest86+ ?

Jul 17 05:34:07 s1 kernel: <3>Debug: sleeping function called from
invalid context at include/linux/rwsem.h:43

Jul 17 05:34:07 s1 kernel:
Jul 17 05:34:07 s1 kernel: Unable to handle kernel paging request at
virtual address 00040000

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"I dream about dreams about dreams", sang the nightingale
under the pale moon (Sandman)

#7Clodoaldo Pinto Neto
clodoaldo_pinto@yahoo.com.br
In reply to: Alvaro Herrera (#6)
Re: server closed the connection unexpectedly

Thank you Alvaro,

Just filled a RMA for the memory that is 20 days old. It only works
with the safest memory settings. Should have done it before. Have never
seem a (very slow) memory test catch bad memory, so won't waste my
time.

Regards,
Clodoaldo Pinto

--- Alvaro Herrera <alvherre@dcc.uchile.cl> escreveu: > On Sun, Jul 18,
2004 at 08:18:07AM -0300, Clodoaldo Pinto Neto wrote:

Thank you for you reply. There is no sign in /var/log/messages of
system crash or reboot when or before the problem happened. Also I
always stop postmaster with pg_ctl stop and there was sometime that

I

last stoped it.

This is clearly a kernel problem. Postgres does not have enough
privileges to trigger this kind of mess.

Or maybe you have bad hardware; RAM, most likely. Have you run
hardware
testing tools, like Memtest86+ ?

Jul 17 05:34:07 s1 kernel: <3>Debug: sleeping function called from
invalid context at include/linux/rwsem.h:43

Jul 17 05:34:07 s1 kernel:
Jul 17 05:34:07 s1 kernel: Unable to handle kernel paging request

at

virtual address 00040000

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"I dream about dreams about dreams", sang the nightingale
under the pale moon (Sandman)

---------------------------(end of
broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

_______________________________________________________
Yahoo! Mail agora com 100MB, anti-spam e antiv�rus gr�tis!
http://br.info.mail.yahoo.com/