Why does my Splunk process on my search head show not available abruptly?

[jkamdar]

I have noticed that Splunk process on my development search-head shows not available abruptly. And then it becomes available. One of those time, I check the status of Splunk service and below is the output. Not sure, where to check and how to troubleshoot this problem. I was just assigned this problem, so not sure how often it happens but I have wrote a cronjob to check the status every 5 minutes and got an alert that it happened last night around 3am. But as far as I know, there is no pattern to it. 

systemctl status splunk 

splunk.service - Splunk Enterprise 

   Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled) 

   Active: deactivating (stop-sigterm) (Result: exit-code) since Mon 2022-10-03 10:45:41 EDT; 41s ago 

 Main PID: 76827 (code=exited, status=8) 

   CGroup: /system.slice/splunk.service 

           └─78229 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py 

Oct 03 10:02:52 splunkdev01.xxx.xxx splunk[76759]: [  OK  ] 

Oct 03 10:02:52 splunkdev01.xxx.xxx splunk[76759]: All installed files intact. 

Oct 03 10:02:52 splunkdev01.xxx.xxx splunk[76759]: Done 

Oct 03 10:02:52 splunkbdev01.xxx.xxx splunk[76759]: All preliminary checks passed. 

Oct 03 10:02:52 splunkdev01.xxx.xxx splunk[76759]: Starting splunk server daemon (splunkd)... 

Oct 03 10:02:52 splunkdev01.xxx.xxx splunk[76759]: Done 

Oct 03 10:04:31 splunkdev01.xxx.xxx sudo[78359]:   splunk : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/netstat -anp 

Oct 03 10:04:31 splunkdev01.xxx.xxx sudo[78359]: pam_unix(sudo:session): session opened for user root by (uid=0) 

Oct 03 10:04:43 splunkdev01.xxx.xxx systemd[1]: Started Splunk Enterprise. 

Oct 03 10:45:41 splunkdev01.xxx.xxx systemd[1]: splunk.service: main process exited, code=exited, status=8/n/a 

[jkamdar]

Good point, I will check it out, thanks. 

[richgalloway]

It's possible the OOM Killer stopped splunkd to recover memory.  Check /var/log/messages.

[PickleRick]

Yes, that's the typical reason on dev environments which are typically low on resources "because there will never be any reasonable utilization". Typical culprit is datamodel acceleration - summary-building searches can utilize quite a lot of memory even if you don't have much other activity.