Splunk Search

Why is my alert not triggered even though there are results that meet the criteria?

kotig
Path Finder

Hi All,

We have a search which checks for a total count of failures in system in the last 24 hours:

index=mydata earliest=-24h| stats list(Summary) as "Summary", count(Summary) as SummaryCount

when a specific failures occur, the SummaryCount has a value of 1 or more than one.

Based on that we have created an alert to trigger with custom trigger condition:
search SummaryCount>=1 and AlertType =Scheduled and is scheduled to run every day, one time at 1 am.
and the Action Options, set was "Once"

Every time we run our search we get a result, so i was expecting there will be 1 alert everyday until the SummaryCount value is returned. But i got the alert only on 1 day and later it does not send any alert emails.

Can you please help? This is really urgent.

Thanks
Koti

0 Karma
1 Solution

kotig
Path Finder

Thank you, appreciate your help on trying to help me. My alert is fixed now. I had a false alarm. Thanks everyone for your inputs.

View solution in original post

0 Karma

kotig
Path Finder

Thank you, appreciate your help on trying to help me. My alert is fixed now. I had a false alarm. Thanks everyone for your inputs.

0 Karma

lguinn2
Legend

I suggest that you should simply use the normal trigger condition of Number of Results > 0
Your custom search for the alert is broken in a couple of ways:

  • and must be capitalized as AND in searches if you want a boolean comparison.
  • Your search does not yield a field named AlertType

What this means is that your custom search will never return anything - so your alert will not fire.
As far as I can tell, you don't need a custom search for your alert at all. Change your settings as
suggested in the first paragraph and see what happens.

0 Karma

somesoni2
Revered Legend

How about you update your alert like this and try.

Search: index=mydata earliest=-24h| stats list(Summary) as "Summary", count(Summary) as SummaryCount | wher SummaryCount>0

Time Range: -1d@d to @d
Alert Type: Scheduled, Cron: 0 1 * * *
Alert condition: If number of events greater than 0
Action Options: Once

0 Karma

kotig
Path Finder

Sure thank you will try that option.

0 Karma

somesoni2
Revered Legend

Did you run the query for days you didn't get alert and found results matching your trigger condition?

0 Karma

kotig
Path Finder

Yes, i am getting value for the day it did not trigger.

0 Karma
Get Updates on the Splunk Community!

Upcoming Webinar: Unmasking Insider Threats with Slunk Enterprise Security’s UEBA

Join us on Wed, Dec 10. at 10AM PST / 1PM EST for a live webinar and demo with Splunk experts! Discover how ...

.conf25 technical session recap of Observability for Gen AI: Monitoring LLM ...

If you’re unfamiliar, .conf is Splunk’s premier event where the Splunk community, customers, partners, and ...

A Season of Skills: New Splunk Courses to Light Up Your Learning Journey

There’s something special about this time of year—maybe it’s the glow of the holidays, maybe it’s the ...