topic Re: Splunk keep shutting down by it self in Alerting

Splunk keep shutting down by it self

Nezar — Mon, 28 Sep 2020 14:48:37 GMT

Hello,

We have Splunk 7.1.1 with 16 CPU and 8G physical memory, It's keep shutting down by its self, At the beginning it was every day and the disk was over 80% so we made the frozenTimePeriodInSec=31536000, and that make the disk stable at 66%. The problem was better for few days but then it returned back with message related to the number of search witch is more 5000 , To fix that we run a code to delete the the empty the Dispatch every 1hour. That's made the search more faster but didn't fix the problem. The memory usage is always more than 70% now and it's very unstable (the Cpu usage is good not more 20%).

Re: Splunk keep shutting down by it self

gcusello — Mon, 28 Sep 2020 14:57:08 GMT

Hi @Nezar,

check the disk space occupation.

Anyway, shut down isn't a correct behavior!

I hint to open a Case to Splunk Support, they can check the configurations using the diag.

Ciao.

Giuseppe

Re: Splunk keep shutting down by it self

richgalloway — Mon, 28 Sep 2020 14:59:45 GMT

Check /var/log/messages for messages from the Out Of Memory (OOM) killer. If you find them then Splunk is being shut down by the OS for using too much memory. The solution is to add more memory or reduce the number of searches being run. Consider adding memory, anyway, since you are below the recommended 12GB.

On the whole, it seems your Splunk environment is outdated and undersized. Think about having PS review the installation.