Splunk indexers crashing

[ajji2684]

Team,

We have added 1800 more forwarders that report very small data (around 100MB all to gether)to Splunk, as soon as we started them , splunk indexers started crashing and they are crashing repeatedly soon after we start.

We are running on AIX and splunk version is 4.3.3
in the crash log am seeing below message

Received fatal signal 11 (Segmentation fault).
Cause:
Memory access denied at address [0x00000004].
Crashing thread: TcpInputProcessor
Registers:
IAR: [0x10031E3C] ?

any help would be appreacieated

[ShaneNewman]

We saw the same thing on our AIX indexer. Every time we restarted the instance it would crash within 5-10 minutes. After dealing with this for about 3 days, we upgraded to 6.0 and that did not help either.

One of my co-workers put in a ticket with Splunk, nothing useful ever came of it though.

At the end of the day, we ended up converting that server from an indexer to a heavy forwarder. It has been up for 2 months now with no issues. We still have it doing all of the extractions and reassigning indexes/sourcetypes based on eventtypes and regex matches, the only change was to stop indexing on that server.

You can always submit your own ticket to Splunk, go to the /opt/Splunk/bin directory and run splunk diag. This will give you a rather large file to upload to your support ticket, if your company does not pay for support... be prepared to wait a few days for a response.

[RishiMandal]

Same here. we kept it idle for one year, just like a dummy server without splunk running. We recently upgraded to 7 version and started hoping the indexer runs smooth, but still the same issue
FATAL ProcessRunner - Unexpected EOF from process runner child!
FATAL ProcessRunner - Unexpected EOF from process runner child!

[gfuente]

Hello

Did you set the server ulimits properly?

http://docs.splunk.com/Documentation/Splunk/6.0/Troubleshooting/ulimitErrors

Regards

[ajji2684]

yes,

Ulimit is set to unlimited