Alerting

Why isn't my splunk alert firing?

ushumar
New Member

Hi,

I'm using Splunk Enterprise 7.1.1 and I'm having a problem with alerts.
I have tried to create an alert for the following search:

host="Alert_Trial"|noop|stats count|eval count = count + 1

I want the alert to send an email and I have configured my email settings as per the following link:
https://www.splunk.com/blog/2014/06/27/splunk-alerts-using-gmail-twitter-phone-calls-and-much-more.h...

I have triggered my alert to fire for Number of events greater than 0 and I have set it for every 5 minutes using Cron Schedule: */5 * * * *

I save the alert and it appears in my alerts list. But no longer how long I wait I never get an e-mail and the alert always says "There are no fired events for this alert". I have also tried using "Real-time" instead of Cron Schedule, but it still does not fire.

Can anyone help me resolve this?

Tags (3)
0 Karma

dkolekar_splunk
Splunk Employee
Splunk Employee

In order to troubleshoot this issue, kindly check below:

  1. Check triggered alerts from
    Settings > Knowledge > Searches, reports and alerts > Alerts column
    OR
    Activity > Triggered alerts

  2. Check under python.log for any error/warning message related to savedsearch/alert you want to trigger

  3. Also, you may need to check for the skipped searches. Maybe during skippedsearches time, you were running into your maxconcurrent limit, which is why this search was skipped multiple times and that is why you did not receive the alert.

Example of the log is as below:

INFO SavedSplunker - savedsearch_id="nobody;SystemManage;SVaccount-authfail-emailsend", user="abcd", app="", savedsearch_name="", priority=, status=skipped, reason="maxconcurrent limit reached", scheduled_time=1498555860, window_time=0

In case if you see the above info message in logs, you should increase the limit for the maximum number of concurrent searches in limits.conf
For reference, kindly check: https://answers.splunk.com/answers/4888/getting-error-maximum-number-of-concurrent-searches-has-been...

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...