I want to set up an alert when an indexer is down. Is there a way to check if an indexer is down from a search head?
It's hard to prove a negative in Splunk and this falls into that category. However, after many failed attempts this seems to be working for me.
| eventcount summarize=false index="<your_index>" OR index="<another_index>"
| dedup index
| fields index
| join type=left index
[| tstats count by index]
| where isnull(count)
| rename index as theindex
You have to list indexes manually and by name with quotes. The last line renames 'index' to 'theindex' as 'index' is a special term in Splunk and will cause problems here if not renamed. I have this run once a day going back 36 hours, this reduces false positives for indexes that may only consume logs once a day.
This is an old question, so I am posting a 2018 answer to mention the Monitoring Console, which has dashboards for indexer performance and status. See the Monitoring Splunk Enterprise documentation.
Absolutely agree with this answer. I add alerts to slack and email when an indexer is down.
Hi! Perhaps I'm a bit late to the party, but are you able to send alerts to Slack from alerts running in the Monitoring Console app?
I have a Slack integration which seemingly works just fine, but for some reason, alerts created inside the Monitoring Console app wont trigger the Slack webhook. I'm wondering if there are some special prerequisites for this specific app.
Gorgeous indexer monitoring - no doubt.
Just had to do this due to some crashing issues we're seeing post 5.0.2 upgrade. This seems to work for us:
index="_internal" source="*splunkd.log" host="your.search.head" "Unable to distribute to peer named" | rex field=_raw "Unable to distribute to peer named (?<indexer>.*):\d+ at " | dedup indexer
I found a way to alert:
index="_internal" source="*splunkd.log" Connect to X.X.X.X:9997 failed | table _time | eval Status= "Down" | eval Indexer = "SplunkIndexerName"
Here X.X.X.X is the ip of the splunk indexer.
It shows the ip in the search but am not able to extract it into my query. So am hardcoding the server name to show it in the results. Is there a way to convert the ip to server name?
What if i want to see for multiple ip of the splunk indexers?
I haven't solved this with Splunk itself. My solution was to add TCP monitors from external monitoring systems to check that 8089 and receiving ports (9997, etc.) are up.