Getting Data In

Monitoring Indexers from Search head

ollie920049
Path Finder

Hi,

Recently I had an error "Some search results may be incomplete or duplicated as we try to fix up your cluster" which was due to an indexer being unavailable.

However, the splunkd and splunkweb services where both running, so none of our service monitoring tools caught it. I would like to tap into the same internal logs that splunk will be using to create that alert in the web dashbaord, and write my own alert that fires us an email when this happens.

Does anybody know what internal logs are of interest to monitor the state of an indexer?

Thanks in advance

Tags (2)
0 Karma

lmyrefelt
Builder

index=_internal contains most of the information and more . as you should find most things logged to splunkd.log .
(There is a new index under index=_interspection ... but i have not checked or played around with it yet.)

You can use the following to help diagnoze your splunk installation:

https://apps.splunk.com/app/1294/

and

https://apps.splunk.com/app/748/

The following can also be usefull:

https://apps.splunk.com/app/1769/
https://apps.splunk.com/app/1848/

But especially the SOS app will give you help in troubleshooting your installation.

you will find the docs here: docs.splunk.com

Good luck!

gfuente
Motivator
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...