topic Re: Splunk indexers crashing in Getting Data In

Splunk indexers crashing

ajji2684 — Wed, 18 Dec 2013 16:23:19 GMT

Team,

We have added 1800 more forwarders that report very small data (around 100MB all to gether)to Splunk, as soon as we started them , splunk indexers started crashing and they are crashing repeatedly soon after we start.

We are running on AIX and splunk version is 4.3.3
in the crash log am seeing below message

Received fatal signal 11 (Segmentation fault).
Cause:
Memory access denied at address [0x00000004].
Crashing thread: TcpInputProcessor
Registers:
IAR: [0x10031E3C] ?

any help would be appreacieated

Re: Splunk indexers crashing

gfuente — Wed, 18 Dec 2013 16:26:17 GMT

Hello

Did you set the server ulimits properly?

http://docs.splunk.com/Documentation/Splunk/6.0/Troubleshooting/ulimitErrors

Regards

Re: Splunk indexers crashing

ajji2684 — Wed, 18 Dec 2013 16:37:02 GMT

yes,

Ulimit is set to unlimited

Re: Splunk indexers crashing

ShaneNewman — Thu, 19 Dec 2013 04:19:15 GMT

We saw the same thing on our AIX indexer. Every time we restarted the instance it would crash within 5-10 minutes. After dealing with this for about 3 days, we upgraded to 6.0 and that did not help either.

One of my co-workers put in a ticket with Splunk, nothing useful ever came of it though.

At the end of the day, we ended up converting that server from an indexer to a heavy forwarder. It has been up for 2 months now with no issues. We still have it doing all of the extractions and reassigning indexes/sourcetypes based on eventtypes and regex matches, the only change was to stop indexing on that server.

You can always submit your own ticket to Splunk, go to the /opt/Splunk/bin directory and run splunk diag. This will give you a rather large file to upload to your support ticket, if your company does not pay for support... be prepared to wait a few days for a response.

Re: Splunk indexers crashing

RishiMandal — Tue, 14 May 2019 21:05:09 GMT

Same here. we kept it idle for one year, just like a dummy server without splunk running. We recently upgraded to 7 version and started hoping the indexer runs smooth, but still the same issue
FATAL ProcessRunner - Unexpected EOF from process runner child!
FATAL ProcessRunner - Unexpected EOF from process runner child!