Monitoring Splunk
Highlighted

Splunkd crashing because of Segmentation fault

Explorer

Hello,

we have running Splunk Version 4.1.6 build 89596 on AIX 6.1. From time to time splunkd is crashing with Segmentation fault on address [0x00000004]. Its always the same address who causes the problem. Here is an extract out of the crash.log:

[build 89596] 2011-01-26 09:52:12
Received fatal signal 11 (Segmentation fault).
Cause:
Memory access denied at address [0x00000004].
Crashing thread: Main Thread
Registers:
IAR: [0x1001EAE4] ?
MSR: [0x0000D032]
R0: [0xFFFFFFFF]
R1: [0x2FF22320]
R2: [0x00000000]
R3: [0x00000001]
R4: [0x432B2B00]
R5: [0x00000008]
R6: [0x00000010]
R7: [0x3453F968]
R8: [0x2FF21534]
...

What can we do? Is this problem known? Where to send the crash.log

Kind regards,

Aaron

Tags (2)
0 Karma
Highlighted

Re: Splunkd crashing because of Segmentation fault

Splunk Employee
Splunk Employee

This is a splunk support issue. Please log a case via the splunk website if you have not done so already. http://www.splunk.com/support

For most platforms, the crash log includes a stack trace. Is that missing in your environment? The registers are pretty hard to go on by themselves.

Support will want to do the normal dance of "when did it start happening, how often, any correlations etc." The information you provide so far just simply tells us that some part of splunk is following a null pointer. It is quite a lot of work to get from there to what is wrong. It is likely not possible without a fair amount more data gathering.

Highlighted

Re: Splunkd crashing because of Segmentation fault

Explorer

Thanks for the hint, I gonna open a support call by splunk support today.

regards,

Aaron

0 Karma
Highlighted

Re: Splunkd crashing because of Segmentation fault

Contributor

I managed to work around this by un-taring the current version of Splunk over the top of the installation.
Running a chown command to make sure the files were all owned by the right user, then starting up again.
Worked for me, hope this can help someone else.

0 Karma