Monitoring Indexers from Search head

[ollie920049]

Hi,

Recently I had an error "Some search results may be incomplete or duplicated as we try to fix up your cluster" which was due to an indexer being unavailable.

However, the splunkd and splunkweb services where both running, so none of our service monitoring tools caught it. I would like to tap into the same internal logs that splunk will be using to create that alert in the web dashbaord, and write my own alert that fires us an email when this happens.

Does anybody know what internal logs are of interest to monitor the state of an indexer?

Thanks in advance

[lmyrefelt]

index=_internal contains most of the information and more . as you should find most things logged to splunkd.log .
(There is a new index under index=_interspection ... but i have not checked or played around with it yet.)

You can use the following to help diagnoze your splunk installation:

https://apps.splunk.com/app/1294/

and

https://apps.splunk.com/app/748/

The following can also be usefull:

https://apps.splunk.com/app/1769/
https://apps.splunk.com/app/1848/

But especially the SOS app will give you help in troubleshooting your installation.

you will find the docs here: docs.splunk.com

Good luck!

[gfuente]

Hello

You can use the REST command and those endpoints:

https://mymaster:mgmt_port/services/cluster/master/searchheads
https://mymaster:mgmt_port/services/cluster/master/indexes

Regards