topic Re: Alert Triggering only once even if set to 'Per Result' in Alerting

Alert Triggering only once even if set to 'Per Result'

ashutoshab — Mon, 08 Apr 2019 09:01:39 GMT

I have created a scheduled alert that looks for results over a time period and if there are events, it has to send an email for every result. This email alert creates a ticket in our ticketing portal.
Incase if there are 10 results in that time period, Splunk should send 10 emails. But instead Splunk triggers only once and send all the results in one single email. This is weird and I am trying to find solution to it.

Below is my Alert Configuration:

Alert Type: Scheduled Run on Cron Schedule
Time Range: last 15 minutes.
Cron Expression: 0,15,30,45 * * * *
Trigger alert when: Number of results > 0
Trigger: For each Result
Throttle: Unchecked.

When the alert triggers, it generates only one alert with the first result and does not trigger anything for the rest. I want to know what I am missing. I see there are 10 results but only one alert.

Re: Alert Triggering only once even if set to 'Per Result'

horsefez — Mon, 08 Apr 2019 10:51:30 GMT

Hey @ashutoshab ,

which Splunk version are you using?
Also can you post the specific stanza from the savedsearches.conf

Re: Alert Triggering only once even if set to 'Per Result'

ashutoshab — Mon, 08 Apr 2019 12:13:20 GMT

I am using Splunk Enterprise 7.2.4

Below is my Stanza

[STANZA NAME]
action.email = 1
action.email.include.results_link = 0
action.email.include.view_link = 0
action.email.mailserver = localhost
action.email.message.alert = {"RANDOM TEXT
}}
action.email.priority = 1
action.email.subject = <RANDOM TEXT>
action.email.to = RANDOM EMAIL ADDRESS
alert.digest_mode = 0
alert.expires = 1h
alert.severity = 4
alert.suppress = 0
alert.track = 1
counttype = number of events
cron_schedule = 0,15,30,45 * * * *
description = RANDOM DESCRIPTION
dispatch.earliest_time = -15m
dispatch.latest_time = now
display.events.type = raw
display.general.type = statistics
display.page.search.mode = verbose
display.page.search.tab = statistics
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = <SEARCH STRING>

Re: Alert Triggering only once even if set to 'Per Result'

ashutoshab — Thu, 02 May 2019 22:40:29 GMT

Waiting for an answer. I feel this might be a bug.

Re: Alert Triggering only once even if set to 'Per Result'

adonio — Fri, 03 May 2019 16:46:52 GMT

what is your search looks like?

Re: Alert Triggering only once even if set to 'Per Result'

anil15694 — Fri, 17 May 2019 06:13:35 GMT

Hey @ashutoshab,

Here you are creating schedule alert with time range 15 min so alert is getting run every 15 min. If you want your alerts should run and send email on every event so you should create real time alerts. Real - time alerts will be useful to monitor events or event patterns as they happen.

You can use this splunk documentation as reference to create real-time alerts.

https://docs.splunk.com/Documentation/Splunk/7.2.6/Alert/DefineRealTimeAlerts

Re: Alert Triggering only once even if set to 'Per Result'

ashutoshab — Mon, 20 May 2019 12:57:49 GMT

Yes, I know that, but for similar scheduled search, for some other query, I receive alert for every event. I mean, it has a different query but similar schedule of 15 mins. If there are 10 events, I receive 10 emails.

Here, it send only 1 email for everything.

Re: Alert Triggering only once even if set to 'Per Result'

ashutoshab — Mon, 20 May 2019 13:01:40 GMT

index=<someIndexName> sourcetype="<someSourceType>" <SomeField>=* | table eventType, sender, headerFrom, recipient{}, toAddresses{}, subject, imposterScore, GUID, messageTime, phishScore, spamScore, quarantineFolder, senderIP, messageID, threatsInfoMap{}.classification, threatsInfoMap{}.threatUrl, threatsInfoMap{}.threatID, threatsInfoMap{}.campaignID, threatsInfoMap{}.threat, threatsInfoMap{}.threatStatus, threatsInfoMap{}.threatTime, threatsInfoMap{}.threatType

Re: Alert Triggering only once even if set to 'Per Result'

marcoscala — Tue, 23 Jul 2019 09:19:07 GMT

I downvoted this post because not pertinent: if you have a scheduled alert it should fire more actions according to the number or results.

Re: Alert Triggering only once even if set to 'Per Result'

marcoscala — Tue, 23 Jul 2019 09:19:09 GMT

I downvoted this post because not pertinent: if you have a scheduled alert it should fire more actions according to the number or results.

Re: Alert Triggering only once even if set to 'Per Result'

snigdhasaxena — Tue, 23 Jul 2019 12:47:39 GMT

Hi @ashutoshab,

Is it possible to make your query generic and share ?

Re: Alert Triggering only once even if set to 'Per Result'

Sujithkumarkb — Tue, 23 Jul 2019 12:59:57 GMT

@ashutoshab ,seems like your throttle alert is on , which will accumulate all the events every 15mins and react at once.
You can try scheduling the alert in real time with throttle disabled.

Else you can try something like below

The other way round can be to add a counter or a variable condition in your alert query like below.
example : | stats phone by state " into the search and create a custom alert trigger such as " eval count = if(search state =received,1,0) |search count =1 (in your case to trigger every event)".

Hope this helps.