topic Re: splunkd crash in Monitoring Splunk

splunkd crash

preben12 — Mon, 09 Dec 2013 07:18:24 GMT

Splunkd has crashed a couple of times now.
The only thing I see in splunkd.log is some log WARNS
eg.

12-09-2013 07:55:09.879 +0100 WARN HttpListener - Socket error from 10.101.11.1
02 while accessing /services/data/indexes: Broken pipe

and a lot of ->
12-09-2013 08:03:17.651 +0100 WARN HttpListener - HTTP active connections down to 330, unthrottling

Is there some configuration that needs to be set different - or ??

Re: splunkd crash

kristian_kolb — Mon, 09 Dec 2013 11:48:12 GMT

have you checked the crash logs, and the splunkd_stderr.log? They may reveal more info.

Re: splunkd crash

preben12 — Mon, 09 Dec 2013 12:12:28 GMT

There is no crash log, and the only resent error I can find in splunkd.log is

12-04-2013 19:00:36.639 +0100 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!

Other that that there is only the WARN's that seems a bit worrying.

best,
Preben

Re: splunkd crash

alacercogitatus — Mon, 09 Dec 2013 12:22:34 GMT

Install S.O.S. Then monitor the buggy system for RAM usage. Certain Operating Systems just kill the process when they run out of Memory, and since Splunk "didn't do it", no event is written.

Re: splunkd crash

preben12 — Mon, 09 Dec 2013 14:02:02 GMT

I tripled the ram on the box - lets see if this helps splunkd from being killed by linux.

I still see the warn logs
12-09-2013 14:44:59.219 +0100 WARN HttpListener - HTTP active connections down to 330, unthrottling

Any hints on the reason for this ??

Best,
Preben

Re: splunkd crash

grijhwani — Mon, 09 Dec 2013 18:36:21 GMT

Instead of throwing resources at it do as the respondent suggested - use some system level diagnostic tools to find the cause. You don't mention operating system distribution or release number.

Re: splunkd crash

bohanlon_splunk — Tue, 26 Jan 2016 14:14:32 GMT

Are these relevant to you?

https://answers.splunk.com/answers/105292/what-is-the-cause-of-these-socket-errors-reported-in-splunkd-log-since-upgrading-to-6-0.html
https://answers.splunk.com/answers/123473/broken-pipe-issues-when-adding-new-data-source-to-splunk-from-gui.html