splunk web got down

[Splunk_U]

Some wired thing got happen into the server. splunkd was running but the splunk web was down. A restart made everything normal. But i need to know why the splunk web was down. Have got nothing suspicious in splunkd.log and web_server.log.

Is there any way to know why the wplunk web was down??

[yannK · Accepted solution]

Check the _internal index for the logs in web_service.log.
Do you see anything prior to the stopping ?

Otherwise, if you are on linux, check the /var/log/messages for any "Out Of Memory / OOM" events, the system can kill a process.

[Splunk_U]

Yeah that I know...but Sos was not installed in the splunk server when the issue got happened...thats why asking that by seeing event_count, total_run_time etc can we understand the memory consumption...or least can we understand the relation between memory consumption with evvent_count or total_run_time?

[yannK]

the sos app and the ps_sos scripts, will show you cpu/memory of expensive searches
http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk

[Splunk_U]

If my search is having event_count=20000 is it eating much memory than the event_count=100???

[Splunk_U]

I have understood that the root cause is Out of memory but is there any process to check the memory consumption by the search in Splunk?

[Splunk_U]

I have checked the messages. I have found that...
sisidsdaemon invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
and after that..
Out of memory: Kill process 3936 (python) score 730 or sacrifice child
So looks like splunk web got down due to out of memeory...