I have 4 indexers that always have the same memory load (monitored through Zabbix). They are usually consistent at around 90% memory free. However, for some reason splunkix04 — one of my indexers — dropped all of its scheduled jobs and its memory is free up to 95%.
I've checked all the monitoring for splunkix04, but everything is good; it's pointing to two search heads...this has happened before, and I did a systemctl to restart Splunk, and the scheduled jobs did catch on to splunkix04 and everything was normal. However, a few days later, splunkix04 dropped its scheduled searches again. splunkix04 has only 11 Splunk helpers, while my other 3 indexers have about 3 dozen.
- What can be done to ensure splunkix04 doesn't drop its scheduled searches?
- When I do a systemctl restart splunk on splunkix04, what is the impact on logging data (I have 3 other indexers)?
Splunk Enterprise 6.5.3
Thanks.
