topic Re: Monitoring Indexers from Search head in Getting Data In

Monitoring Indexers from Search head

ollie920049 — Fri, 28 Nov 2014 11:27:42 GMT

Hi,

Recently I had an error "Some search results may be incomplete or duplicated as we try to fix up your cluster" which was due to an indexer being unavailable.

However, the splunkd and splunkweb services where both running, so none of our service monitoring tools caught it. I would like to tap into the same internal logs that splunk will be using to create that alert in the web dashbaord, and write my own alert that fires us an email when this happens.

Does anybody know what internal logs are of interest to monitor the state of an indexer?

Thanks in advance

Re: Monitoring Indexers from Search head

gfuente — Fri, 28 Nov 2014 11:36:51 GMT

Hello

You can use the REST command and those endpoints:

https://mymaster:mgmt_port/services/cluster/master/searchheads
https://mymaster:mgmt_port/services/cluster/master/indexes

Regards

Re: Monitoring Indexers from Search head

lmyrefelt — Mon, 28 Sep 2020 18:17:54 GMT

index=_internal contains most of the information and more . as you should find most things logged to splunkd.log .
(There is a new index under index=_interspection ... but i have not checked or played around with it yet.)

You can use the following to help diagnoze your splunk installation:

https://apps.splunk.com/app/1294/

and

https://apps.splunk.com/app/748/

The following can also be usefull:

https://apps.splunk.com/app/1769/
https://apps.splunk.com/app/1848/

But especially the SOS app will give you help in troubleshooting your installation.

you will find the docs here: docs.splunk.com

Good luck!