Alerting

Alert when indexer is down

pdash
Path Finder

I want to set up an alert when an indexer is down. Is there a way to check if an indexer is down from a search head?

Tags (1)
0 Karma

chooglin
Loves-to-Learn

It's hard to prove a negative in Splunk and this falls into that category.  However, after many failed attempts this seems to be working for me.

| eventcount summarize=false index="<your_index>" OR index="<another_index>" 
| dedup index
| fields index
| join type=left index
      [| tstats count by index]
| where isnull(count)
| rename index as theindex

 

You have to list indexes manually and by name with quotes.  The last line renames 'index' to 'theindex' as 'index' is a special term in Splunk and will cause problems here if not renamed.  I have this run once a day going back 36 hours, this reduces false positives for indexes that may only consume logs once a day.

 

0 Karma

ChrisG
Splunk Employee
Splunk Employee

This is an old question, so I am posting a 2018 answer to mention the Monitoring Console, which has dashboards for indexer performance and status. See the Monitoring Splunk Enterprise documentation.

burwell
SplunkTrust
SplunkTrust

Absolutely agree with this answer. I add alerts to slack and email when an indexer is down.

0 Karma

hettervi
Builder

Hi! Perhaps I'm a bit late to the party, but are you able to send alerts to Slack from alerts running in the Monitoring Console app?

I have a Slack integration which seemingly works just fine, but for some reason, alerts created inside the Monitoring Console app wont trigger the Slack webhook. I'm wondering if there are some special prerequisites for this specific app.

0 Karma

ddrillic
Ultra Champion

Gorgeous indexer monitoring - no doubt.

0 Karma

tmeader
Contributor

Just had to do this due to some crashing issues we're seeing post 5.0.2 upgrade. This seems to work for us:

index="_internal" source="*splunkd.log" host="your.search.head" "Unable to distribute to peer named" | rex field=_raw "Unable to distribute to peer named (?<indexer>.*):\d+ at " | dedup indexer

pdash
Path Finder

I found a way to alert:

index="_internal" source="*splunkd.log" Connect to X.X.X.X:9997 failed | table _time | eval Status= "Down" | eval Indexer = "SplunkIndexerName"

Here X.X.X.X is the ip of the splunk indexer.

It shows the ip in the search but am not able to extract it into my query. So am hardcoding the server name to show it in the results. Is there a way to convert the ip to server name?

Amulya888
Explorer

What if i want to see for multiple ip of the splunk indexers?

0 Karma

jspears
Communicator

I haven't solved this with Splunk itself. My solution was to add TCP monitors from external monitoring systems to check that 8089 and receiving ports (9997, etc.) are up.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...