Recently I had an error "Some search results may be incomplete or duplicated as we try to fix up your cluster" which was due to an indexer being unavailable.
However, the splunkd and splunkweb services where both running, so none of our service monitoring tools caught it. I would like to tap into the same internal logs that splunk will be using to create that alert in the web dashbaord, and write my own alert that fires us an email when this happens.
Does anybody know what internal logs are of interest to monitor the state of an indexer?
index=_internal contains most of the information and more . as you should find most things logged to splunkd.log .
(There is a new index under index=_interspection ... but i have not checked or played around with it yet.)
You can use the following to help diagnoze your splunk installation: