Polecat "quit unexpectdly"

Started by Robert Creagerover 14 years ago7 messages
#1Robert Creager
Robert.Creager@Oracle.com
2 attachment(s)

Stack trace, nothing else. Now that the horse has left the barn, I changed to keep error builds...

Process: postgres [17457]
Path: /Volumes/High Usage/usr/local/src/build-farm-4.4/builds/HEAD/pgsql.6849/src/test/regress/tmp_check/install/usr/local/src/build-farm-4.4/builds/HEAD/inst/bin/postgres
Identifier: postgres
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: postgres [17136]

Date/Time: 2011-06-14 05:00:27.466 -0600
OS Version: Mac OS X 10.6.7 (10J869)
Report Version: 6

Interval Since Last Report: 658000 sec
Crashes Since Last Report: 1
Per-App Crashes Since Last Report: 1
Anonymous UUID: 77053C10-A2B5-4078-A796-5862E233A1AC

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Application Specific Information:
abort() called

Thread 0 Crashed: Dispatch queue: com.apple.main-thread
0 libSystem.B.dylib 0x00007fff821815d6 __kill + 10
1 libSystem.B.dylib 0x00007fff82221cd6 abort + 83
2 postgres 0x00000001002fcd52 0x100000000 + 3132754
3 postgres 0x000000010005cafa multixact_twophase_postcommit + 74 (multixact.c:1367)
4 postgres 0x000000010005deab ProcessRecords + 91 (twophase.c:1407)
5 postgres 0x000000010005f78a FinishPreparedTransaction + 1610 (twophase.c:1368)
6 postgres 0x0000000100236a5a PortalRunUtility + 298 (palloc.h:88)
7 postgres 0x00000001002381c5 PortalRunMulti + 597 (pquery.c:1315)
8 postgres 0x0000000100238b78 PortalRun + 984 (pquery.c:817)
9 postgres 0x0000000100234b2d exec_simple_query + 589 (postgres.c:1025)
10 postgres 0x00000001002356f1 PostgresMain + 1937 (postgres.c:3926)
11 postgres 0x00000001001e82ec ServerLoop + 2780 (postmaster.c:3600)
12 postgres 0x00000001001e9251 PostmasterMain + 2753 (postmaster.c:1120)
13 postgres 0x000000010017c305 main + 965 (main.c:199)
14 postgres 0x0000000100000b14 start + 52

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x0000000000000018 rcx: 0x00007fff5fbfd3b8 rdx: 0x0000000000000000
rdi: 0x0000000000004431 rsi: 0x0000000000000006 rbp: 0x00007fff5fbfd3d0 rsp: 0x00007fff5fbfd3b8
r8: 0x0000000000000004 r9: 0x0000000000000000 r10: 0x00007fff8217d616 r11: 0xffffff80002e28b0
r12: 0x000000000000006a r13: 0x0000000100508980 r14: 0x0000000000000aba r15: 0x00000001004ffb20
rip: 0x00007fff821815d6 rfl: 0x0000000000000202 cr2: 0x00007fff70419230

Binary Images:
0x100000000 - 0x1004fbfef +postgres ??? (???) <E4B56AD5-5ABD-37F5-8D5A-ADE9F24372F2> /Volumes/High Usage/usr/local/src/build-farm-4.4/builds/HEAD/pgsql.6849/src/test/regress/tmp_check/install/usr/local/src/build-farm-4.4/builds/HEAD/inst/bin/postgres
0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???) <486E6C61-1197-CC7C-2197-82CE505102D7> /usr/lib/dyld
0x7fff80b92000 - 0x7fff80cb3fe7 libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <48AEAFE1-21F4-B3C8-4199-35AD5E8D0613> /usr/lib/libcrypto.0.9.8.dylib
0x7fff80cb4000 - 0x7fff80cc5ff7 libz.1.dylib 1.2.3 (compatibility 1.0.0) <97019C74-161A-3488-41EC-A6CA8738418C> /usr/lib/libz.1.dylib
0x7fff80df1000 - 0x7fff80f08fef libxml2.2.dylib 10.3.0 (compatibility 10.0.0) <1B27AFDD-DF87-2009-170E-C129E1572E8B> /usr/lib/libxml2.2.dylib
0x7fff82132000 - 0x7fff822f3fff libSystem.B.dylib 125.2.10 (compatibility 1.0.0) <9BAEB2F2-B485-6349-E1AB-637FE12EE770> /usr/lib/libSystem.B.dylib
0x7fff83300000 - 0x7fff834befff libicucore.A.dylib 40.0.0 (compatibility 1.0.0) <2C6ECACF-CD56-1714-6F63-CB6F5EE7A1E2> /usr/lib/libicucore.A.dylib
0x7fff835a7000 - 0x7fff835b9fe7 libsasl2.2.dylib 3.15.0 (compatibility 3.0.0) <76B83C8D-8EFE-4467-0F75-275648AFED97> /usr/lib/libsasl2.2.dylib
0x7fff83c6c000 - 0x7fff83c70ff7 libmathCommon.A.dylib 315.0.0 (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> /usr/lib/system/libmathCommon.A.dylib
0x7fff843c3000 - 0x7fff843c4ff7 com.apple.TrustEvaluationAgent 1.1 (1) <5952A9FA-BC2B-16EF-91A7-43902A5C07B6> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
0x7fff85c2e000 - 0x7fff85cabfef libstdc++.6.dylib 7.9.0 (compatibility 7.0.0) <35ECA411-2C08-FD7D-11B1-1B7A04921A5C> /usr/lib/libstdc++.6.dylib
0x7fff85d27000 - 0x7fff85d48fff libresolv.9.dylib 41.0.0 (compatibility 1.0.0) <9F322F47-0584-CB7D-5B73-9EBD670851CD> /usr/lib/libresolv.9.dylib
0x7fff85e5c000 - 0x7fff85e99fff com.apple.LDAPFramework 2.0 (120.1) <54A6769E-D7E2-DBE2-EA61-87B9EA355DA4> /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
0x7fff87cdd000 - 0x7fff87d17fff libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <C7153747-50E3-32DA-426F-CC4C505D1D6C> /usr/lib/libssl.0.9.8.dylib
0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???) <9BAEB2F2-B485-6349-E1AB-637FE12EE770> /usr/lib/libSystem.B.dylib

Model: MacBookPro5,1, BootROM MBP51.007E.B05, 2 processors, Intel Core 2 Duo, 2.53 GHz, 6 GB, SMC 1.33f8
Graphics: NVIDIA GeForce 9600M GT, NVIDIA GeForce 9600M GT, PCIe, 512 MB
Graphics: NVIDIA GeForce 9400M, NVIDIA GeForce 9400M, PCI, 256 MB
Memory Module: global_name
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8D), Broadcom BCM43xx 1.0 (5.10.131.36.9)
Bluetooth: Version 2.4.0f1, 2 service, 19 devices, 1 incoming serial ports
Network Service: Ethernet, Ethernet, en0
Network Service: AirPort, AirPort, en1
Network Service: Parallels Shared Networking Adapter, Ethernet, en3
Network Service: Parallels Host-Only Networking Adapter, Ethernet, en4
Serial ATA Device: OWC Mercury Extreme Pro SSD, 223.57 GB
Serial ATA Device: MATSHITADVD-R UJ-868
USB Device: Hub, 0x0409 (NEC Corporation), 0x005a, 0x24100000
USB Device: Hub, 0x0409 (NEC Corporation), 0x005a, 0x24140000
USB Device: Hub, 0x0409 (NEC Corporation), 0x005a, 0x24142000
USB Device: TOSHIBA Video Dock, 0x17e9 (DisplayLink (UK) Ltd.), 0x0110, 0x24142100
USB Device: USB Sound Device, 0x0d8c (C-MEDIA ELECTRONICS INC.), 0x0102, 0x24142300
USB Device: Hub, 0x0409 (NEC Corporation), 0x005a, 0x24142400
USB Device: Keyboard Hub, 0x05ac (Apple Inc.), 0x1006, 0x24142420
USB Device: Apple Keyboard, 0x05ac (Apple Inc.), 0x0220, 0x24142422
USB Device: Hub, 0x0424 (SMSC), 0x2514, 0x24142410
USB Device: My Passport 070A, 0x1058 (Western Digital Technologies, Inc.), 0x070a, 0x24142440
USB Device: AX88772, 0x0b95 (ASIX Electronics Corporation), 0x7720, 0x24142200
USB Device: OWC Express USB3.0, 0x0dc4 (Macpower & Tytech Technology Co., LTD.), 0x020a, 0x24144000
USB Device: ST94811U2-RK, 0x0bc2, 0x0502, 0x24143000
USB Device: Built-in iSight, 0x05ac (Apple Inc.), 0x8507, 0x24400000
USB Device: BRCM2046 Hub, 0x0a5c (Broadcom Corp.), 0x4500, 0x06100000
USB Device: Bluetooth USB Host Controller, 0x05ac (Apple Inc.), 0x8213, 0x06110000
USB Device: Apple Internal Keyboard / Trackpad, 0x05ac (Apple Inc.), 0x0236, 0x04600000
USB Device: IR Receiver, 0x05ac (Apple Inc.), 0x8242, 0x04500000

--

Robert Creager, Principal Software Engineer
Oracle Server Technologies
500 Eldorado Blvd, Bldg 5
Broomfield, CO, 80021
Phone: 303-272-6830
Email: Robert.Creager@Oracle.com

Oracle is committed to developing practices and products that help protect the environment

Attachments:

oracle_sig_logo.gifimage/gif; name=oracle_sig_logo.gifDownload
GIF89ar������������00�``�  ����PP�������pp����@@�������!�,r� $�di�h��l��p,�t-
�����p�Xt�� 7�9���h� ��(Pxx�`p�'b��^� E���A
8�
��@�"[m}gdf�_Z%	b'�Xx-s��
k�����e^	���q�`%����?wy+_������*����(n&
x�ps�,��)�^�1|R-���*_%
_�[_��H�B����������8����A�C�\T� M�� C~|��p�>@e�;�� ��2�@Y�U	^|�@��ARg�� ��DBK%h�����j$p9R��4
���,�O����bA�s��i���Ml��v�5:�PJEGP/X]�E�$������@��/R(�Y�%���(b8���J`�I&q���B���m�zQ���!5�#�JjF�%�7��{M��������@`�N�D�D@�������Q�@��$@�d��E�� ~�m�������(��h��&���6���F(��
;
green-for-email-sig_0.gifimage/gif; name=green-for-email-sig_0.gifDownload
#2Stefan Kaltenbrunner
stefan@kaltenbrunner.cc
In reply to: Robert Creager (#1)
Re: [Pgbuildfarm-members] Polecat "quit unexpectdly"

On 06/14/2011 06:45 PM, Robert Creager wrote:

Stack trace, nothing else. Now that the horse has left the barn, I changed to keep error builds...

Process: postgres [17457]
Path: /Volumes/High Usage/usr/local/src/build-farm-4.4/builds/HEAD/pgsql.6849/src/test/regress/tmp_check/install/usr/local/src/build-farm-4.4/builds/HEAD/inst/bin/postgres
Identifier: postgres
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: postgres [17136]

Date/Time: 2011-06-14 05:00:27.466 -0600
OS Version: Mac OS X 10.6.7 (10J869)
Report Version: 6

Interval Since Last Report: 658000 sec
Crashes Since Last Report: 1
Per-App Crashes Since Last Report: 1
Anonymous UUID: 77053C10-A2B5-4078-A796-5862E233A1AC

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Application Specific Information:
abort() called

Thread 0 Crashed: Dispatch queue: com.apple.main-thread
0 libSystem.B.dylib 0x00007fff821815d6 __kill + 10
1 libSystem.B.dylib 0x00007fff82221cd6 abort + 83
2 postgres 0x00000001002fcd52 0x100000000 + 3132754
3 postgres 0x000000010005cafa multixact_twophase_postcommit + 74 (multixact.c:1367)
4 postgres 0x000000010005deab ProcessRecords + 91 (twophase.c:1407)
5 postgres 0x000000010005f78a FinishPreparedTransaction + 1610 (twophase.c:1368)
6 postgres 0x0000000100236a5a PortalRunUtility + 298 (palloc.h:88)
7 postgres 0x00000001002381c5 PortalRunMulti + 597 (pquery.c:1315)
8 postgres 0x0000000100238b78 PortalRun + 984 (pquery.c:817)
9 postgres 0x0000000100234b2d exec_simple_query + 589 (postgres.c:1025)
10 postgres 0x00000001002356f1 PostgresMain + 1937 (postgres.c:3926)
11 postgres 0x00000001001e82ec ServerLoop + 2780 (postmaster.c:3600)
12 postgres 0x00000001001e9251 PostmasterMain + 2753 (postmaster.c:1120)
13 postgres 0x000000010017c305 main + 965 (main.c:199)
14 postgres 0x0000000100000b14 start + 52

this looks like it was caused by:

http://git.postgresql.org/gitweb?p=postgresql.git;a=commit;h=b81831acbc671445061ed41a55fb1cc21d8e2979

but should now be fixed by:

http://git.postgresql.org/gitweb?p=postgresql.git;a=commit;h=85ea93384ae21ff59f5e5b292884a86f9c10b852

Stefan

#3Kevin Grittner
Kevin.Grittner@wicourts.gov
In reply to: Robert Creager (#1)
Re: Polecat "quit unexpectdly"

Robert Creager <Robert.Creager@Oracle.com> wrote:

Stack trace, nothing else.

3 postgres 0x000000010005cafa
multixact_twophase_postcommit + 74 (multixact.c:1367)
4 postgres 0x000000010005deab
ProcessRecords + 91 (twophase.c:1407)
5 postgres 0x000000010005f78a
FinishPreparedTransaction + 1610 (twophase.c:1368)

If this was a checkout from more than about 7 hours ago and less
than about 10 hours ago, please get a fresh copy of the source and
try again.

-Kevin

#4Robert Creager
Robert.Creager@Oracle.com
In reply to: Kevin Grittner (#3)
2 attachment(s)
Re: Polecat "quit unexpectdly"

On Jun 14, 2011, at 2:11 PM, Kevin Grittner wrote:

Robert Creager <Robert.Creager@Oracle.com> wrote:

Stack trace, nothing else.

3 postgres 0x000000010005cafa
multixact_twophase_postcommit + 74 (multixact.c:1367)
4 postgres 0x000000010005deab
ProcessRecords + 91 (twophase.c:1407)
5 postgres 0x000000010005f78a
FinishPreparedTransaction + 1610 (twophase.c:1368)

If this was a checkout from more than about 7 hours ago and less
than about 10 hours ago, please get a fresh copy of the source and
try again.

You believe it was related to the flurry of errors that popped up then. This machine updates git every 30 minutes, runs builds every 2 hours, forces HEAD every 6 hours.

Later,
Rob

--

Attachments:

oracle_sig_logo.gifimage/gif; name=oracle_sig_logo.gifDownload
GIF89ar������������00�``�  ����PP�������pp����@@�������!�,r� $�di�h��l��p,�t-
�����p�Xt�� 7�9���h� ��(Pxx�`p�'b��^� E���A
8�
��@�"[m}gdf�_Z%	b'�Xx-s��
k�����e^	���q�`%����?wy+_������*����(n&
x�ps�,��)�^�1|R-���*_%
_�[_��H�B����������8����A�C�\T� M�� C~|��p�>@e�;�� ��2�@Y�U	^|�@��ARg�� ��DBK%h�����j$p9R��4
���,�O����bA�s��i���Ml��v�5:�PJEGP/X]�E�$������@��/R(�Y�%���(b8���J`�I&q���B���m�zQ���!5�#�JjF�%�7��{M��������@`�N�D�D@�������Q�@��$@�d��E�� ~�m�������(��h��&���6���F(��
;
green-for-email-sig_0.gifimage/gif; name=green-for-email-sig_0.gifDownload
#5Kevin Grittner
Kevin.Grittner@wicourts.gov
In reply to: Robert Creager (#4)
Re: Polecat "quit unexpectdly"

Robert Creager <Robert.Creager@Oracle.com> wrote:

You believe it was related to the flurry of errors that popped up
then.

I haven't looked at all the error in the "flurry". I think your
particular report is consistent with being caused by this commit:

http://git.postgresql.org/gitweb?p=postgresql.git;a=commit;h=b81831acbc671445061ed41a55fb1cc21d8e2979

and solved by this one:

http://git.postgresql.org/gitweb?p=postgresql.git;a=commit;h=85ea93384ae21ff59f5e5b292884a86f9c10b852

I suspect that other builds based on checkouts in this window of
time would probably show similar errors.

-Kevin

#6Andrew Dunstan
andrew@dunslane.net
In reply to: Robert Creager (#4)
Re: [Pgbuildfarm-members] [HACKERS] Polecat "quit unexpectdly"

On 06/14/2011 04:29 PM, Robert Creager wrote:

This machine updates git every 30 minutes, runs builds every 2 hours, forces HEAD every 6 hours.

Robert,

Every 6 hours is too often - please dial that down. 24 hours should be
plenty.

Also, when you report a problem, please only email the buildfarm list if
it's a problem with the buildfarm software. If it's a Postgres problem
(such as a database crash) please just email -hackers.

Thanks.

cheers

andrew

#7Robert Creager
Robert.Creager@Oracle.com
In reply to: Andrew Dunstan (#6)
Re: [Pgbuildfarm-members] [HACKERS] Polecat "quit unexpectdly"

On Jun 14, 2011, at 2:54 PM, Andrew Dunstan wrote:

On 06/14/2011 04:29 PM, Robert Creager wrote:

This machine updates git every 30 minutes, runs builds every 2 hours, forces HEAD every 6 hours.

Every 6 hours is too often - please dial that down. 24 hours should be plenty.

I had bumped it up to 6 hours from the default because of the perl hang that was seen. I'll take it back to 24 hours.

Also, when you report a problem, please only email the buildfarm list if it's a problem with the buildfarm software. If it's a Postgres problem (such as a database crash) please just email -hackers.

OK.

Later,
Rob
--