Alerting

Why is my alert not getting triggered at the right time?

nivedita_viswan
Path Finder

I have configured a scheduled search, which runs and triggers an alert if the search returns >0 results. This search is scheduled to run every 30 minutes. When the alert is triggered, it sends an email and runs a script.
Once the alert is triggered, I want it to be throttled for the rest of the day. So the next day, from 12:00 AM on-wards, the throttle condition needs to be reset and the alert should be again triggered if and when the alert condition is met.

To implement this, I have set

alert.suppress = 1
alert.suppress.period = 1d

My understanding is, that '1d' should throttle the alert for the remaining of that day. However, I often find that the scheduled search returns a result, say at 4PM, while my alert is only triggered during a later search in the day (say 6:30PM). Is this due to the throttling that may have been introduced during the previous day's triggered alert?

When I check scheduler.log, i see the following entry:

sid="scheduler__xxx__search__RMD5623ae5e44f926665_at_1435674600_314", suppressed=1, thread_id="AlertNotifierWorker-0" 06-30-2015 07:45:01.593 -0700 INFO  SavedSplunker - savedsearch_id="nobody;search;<SearchName>", user="xxx", app="search", savedsearch_name="<Search Name>", status=success, digest_mode=1, scheduled_time=1435675500, dispatch_time=1435675501, run_time=0.250, result_count=1, alert_actions=""

I see suppressed=1 even when result_count=1 during the first scheduled search of the day.

I know that the scheduled search itself doesn't have any errors, since the alert is triggered (email sent and script run) as expected, at the right time, on some days. Am I understanding it wrong? How do I ensure that the alert is always triggered the first time the scheduled search returns a result, on any day?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Setting the period to 1d is the same as setting it to 24h or 1440m or 86400s - one day, not "end of day".

Consider setting up a field that contains the date and use that as your throttling field with a period of 1d. This should suppress events with the same field value, ie the same date - until midnight.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Setting the period to 1d is the same as setting it to 24h or 1440m or 86400s - one day, not "end of day".

Consider setting up a field that contains the date and use that as your throttling field with a period of 1d. This should suppress events with the same field value, ie the same date - until midnight.

nivedita_viswan
Path Finder

Thank you, I will try out your suggestion and then accept your answer.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...