Alerting

Alert when indexer is down

Path Finder

I want to set up an alert when an indexer is down. Is there a way to check if an indexer is down from a search head?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

This is an old question, so I am posting a 2018 answer to mention the Monitoring Console, which has dashboards for indexer performance and status. See the Monitoring Splunk Enterprise documentation.

SplunkTrust
SplunkTrust

Absolutely agree with this answer. I add alerts to slack and email when an indexer is down.

0 Karma

Builder

Hi! Perhaps I'm a bit late to the party, but are you able to send alerts to Slack from alerts running in the Monitoring Console app?

I have a Slack integration which seemingly works just fine, but for some reason, alerts created inside the Monitoring Console app wont trigger the Slack webhook. I'm wondering if there are some special prerequisites for this specific app.

0 Karma

Ultra Champion

Gorgeous indexer monitoring - no doubt.

0 Karma

Contributor

Just had to do this due to some crashing issues we're seeing post 5.0.2 upgrade. This seems to work for us:

index="_internal" source="*splunkd.log" host="your.search.head" "Unable to distribute to peer named" | rex field=_raw "Unable to distribute to peer named (?<indexer>.*):\d+ at " | dedup indexer

Path Finder

I found a way to alert:

index="_internal" source="*splunkd.log" Connect to X.X.X.X:9997 failed | table _time | eval Status= "Down" | eval Indexer = "SplunkIndexerName"

Here X.X.X.X is the ip of the splunk indexer.

It shows the ip in the search but am not able to extract it into my query. So am hardcoding the server name to show it in the results. Is there a way to convert the ip to server name?

Explorer

What if i want to see for multiple ip of the splunk indexers?

0 Karma

Communicator

I haven't solved this with Splunk itself. My solution was to add TCP monitors from external monitoring systems to check that 8089 and receiving ports (9997, etc.) are up.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!