Getting Data In

Monitoring Indexers from Search head

ollie920049
Path Finder

Hi,

Recently I had an error "Some search results may be incomplete or duplicated as we try to fix up your cluster" which was due to an indexer being unavailable.

However, the splunkd and splunkweb services where both running, so none of our service monitoring tools caught it. I would like to tap into the same internal logs that splunk will be using to create that alert in the web dashbaord, and write my own alert that fires us an email when this happens.

Does anybody know what internal logs are of interest to monitor the state of an indexer?

Thanks in advance

Tags (2)
0 Karma

lmyrefelt
Builder

index=_internal contains most of the information and more . as you should find most things logged to splunkd.log .
(There is a new index under index=_interspection ... but i have not checked or played around with it yet.)

You can use the following to help diagnoze your splunk installation:

https://apps.splunk.com/app/1294/

and

https://apps.splunk.com/app/748/

The following can also be usefull:

https://apps.splunk.com/app/1769/
https://apps.splunk.com/app/1848/

But especially the SOS app will give you help in troubleshooting your installation.

you will find the docs here: docs.splunk.com

Good luck!

gfuente
Motivator
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Make Your Machine Data AI-Ready in Splunk

Every team is getting ready for AI agents.    Watch On-Demand   In this Tech Talk, Splunk platform specialists ...

How to Break a Kubernetes Pod on Purpose and Let the AI Troubleshooting Agent Find ...

Let's say I've just rolled out a Kubernetes Deployment with a typo in the image tag. Nothing explodes. The pod ...

Splunk Add-on for Microsoft Windows 11.0.3: Parse Once, Search Faster

The updated Splunk Add-on for Microsoft Windows helps customers search Windows events faster by reducing ...