Splunk keep shutting down by it self

[Nezar]

Hello, 

We have Splunk 7.1.1 with 16 CPU and 8G physical memory, It's keep shutting down by its self, At the beginning it was every day and the disk was over 80% so  we made the frozenTimePeriodInSec=31536000, and that make the disk stable at 66%. The problem was better for few days but then it  returned back with message related to the number of search witch is more 5000 , To fix that we run a code to delete the the empty the Dispatch every 1hour. That's made the search more faster  but didn't fix the problem. The memory usage is always more than 70% now  and it's very unstable (the Cpu usage is good not more 20%).

[richgalloway]

Check /var/log/messages for messages from the Out Of Memory (OOM) killer.  If you find them then Splunk is being shut down by the OS for using too much memory.  The solution is to add more memory or reduce the number of searches being run.  Consider adding memory, anyway, since you are below the recommended 12GB.

On the whole, it seems your Splunk environment is outdated and undersized.  Think about having PS review the installation.

[gcusello]

Hi @Nezar,

check the disk space occupation.

Anyway, shut down isn't a correct behavior!

I hint to open a Case to Splunk Support, they can check the configurations using the diag.

Ciao.

Giuseppe