Why do forwarders go down with Unexpected EOF messages such as "FATAL ProcessRunner - Unexpected EOF from process runner child"?

[ddrillic]

Three of our forwarders went down today saying -

11-17-2016 15:39:33.525 -0600 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.123.200.143_8089_dbslt0080.uhc.com_dbslt0080_DC816E54-5F68-4259-BBE3-A97F24AD2EA8
11-17-2016 15:39:35.015 -0600 FATAL ProcessRunner - Unexpected EOF from process runner child!
11-17-2016 15:39:35.015 -0600 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!

What can the cause be?

[vincenteous]

Hi ddrillic ,

What OS are your Splunk running on top of? Is there any server management software running on the same server? I've had a similar experience like this when my Splunk was running on top of Red Hat with heavy load. If you are running Linux too, you may check from dmesg or /var/log/messages to see whether the similar type of error is recorded there (about terminating a process). If it exists, you may want to recheck the policies of your OS regarding process management.

I hope this could become your reference for troubleshooting.

[kannu]

works for me ,
In my case some other person have killed the process (splunk) through ansible script .