Alerting

Why is my alert not getting triggered at the right time?

nivedita_viswan
Path Finder

I have configured a scheduled search, which runs and triggers an alert if the search returns >0 results. This search is scheduled to run every 30 minutes. When the alert is triggered, it sends an email and runs a script.
Once the alert is triggered, I want it to be throttled for the rest of the day. So the next day, from 12:00 AM on-wards, the throttle condition needs to be reset and the alert should be again triggered if and when the alert condition is met.

To implement this, I have set

alert.suppress = 1
alert.suppress.period = 1d

My understanding is, that '1d' should throttle the alert for the remaining of that day. However, I often find that the scheduled search returns a result, say at 4PM, while my alert is only triggered during a later search in the day (say 6:30PM). Is this due to the throttling that may have been introduced during the previous day's triggered alert?

When I check scheduler.log, i see the following entry:

sid="scheduler__xxx__search__RMD5623ae5e44f926665_at_1435674600_314", suppressed=1, thread_id="AlertNotifierWorker-0" 06-30-2015 07:45:01.593 -0700 INFO  SavedSplunker - savedsearch_id="nobody;search;<SearchName>", user="xxx", app="search", savedsearch_name="<Search Name>", status=success, digest_mode=1, scheduled_time=1435675500, dispatch_time=1435675501, run_time=0.250, result_count=1, alert_actions=""

I see suppressed=1 even when result_count=1 during the first scheduled search of the day.

I know that the scheduled search itself doesn't have any errors, since the alert is triggered (email sent and script run) as expected, at the right time, on some days. Am I understanding it wrong? How do I ensure that the alert is always triggered the first time the scheduled search returns a result, on any day?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Setting the period to 1d is the same as setting it to 24h or 1440m or 86400s - one day, not "end of day".

Consider setting up a field that contains the date and use that as your throttling field with a period of 1d. This should suppress events with the same field value, ie the same date - until midnight.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Setting the period to 1d is the same as setting it to 24h or 1440m or 86400s - one day, not "end of day".

Consider setting up a field that contains the date and use that as your throttling field with a period of 1d. This should suppress events with the same field value, ie the same date - until midnight.

nivedita_viswan
Path Finder

Thank you, I will try out your suggestion and then accept your answer.

0 Karma
Get Updates on the Splunk Community!

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...