topic Re: Triggered Alerts are Occuring twice on every Event that happens once in Alerting

Triggered Alerts are Occuring twice on every Event that happens once

sbeamro — Mon, 08 Dec 2014 09:29:34 GMT

I have configured an Alert that is running in real time.
with the value of host="10.56.183.0" "%LINEPROTO-5-UPDOWN"
since 10.56.183.0 is a switch and I'd like to recieve an email when interface goes up or down.

When the event occurs once (I can see in the search that it ocurrs once) I'm getting 2 emails.
and when looking at the Alert window - I can see that it counted 2.

any idea why ?

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Mon, 11 Jan 2016 20:19:21 GMT

I'm seeing the same behavior with an alert that I configured. The alert is triggered twice but the event only happens once. To be safe I deleted and re-created the alert but the problem didn't go away. Is this an issue with Splunk alerts?

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 00:59:49 GMT

Hi @dschnabel,
Could you share more of your alert configuration? For example, what do you have for the query, the triggering condition, and any throttling? Also, what software version are you using?
Thanks for the details!

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Tue, 29 Sep 2020 08:24:38 GMT

Hi @frobinson,

thanks for your reply. This is the configuration of my alert:

Query: index=tv-* ClientId NOT (some-id-1 OR some-id-2 OR some-id-3) ClientId="*" Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

Trigger Condition: Per-Result

Alert Type: Real-time

No throttling.

Where would I find the software version?

Daniel

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 01:15:52 GMT

Hi Daniel,
You can try the "About" link at bottom-left of a page in Splunk Web:
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Troubleshooting/CheckSplunkversion

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Tue, 12 Jan 2016 01:21:02 GMT

Splunk Version 6.3.1511.1
Splunk Build 90ea9ab275dc
List of Products: retention
Server Name ip-192-168-92-140
[...]
Current Application: Search & Reporting
App Version 6.3.1511.1

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 01:29:33 GMT

That being said, it might be generally useful (regardless of version) to consider throttling the alert. Real-time alerts with per-result triggering can sometimes fire more often than you need. For example, our docs have this note about this alert type:

"You can also use transforming commands to return results based on processing the retrieved events. A per-result alert triggers in both cases, when the search returns an event or when a transforming command returns results."

http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/Defineper-resultalerts

Although I don't see a transforming command in your query example, there might be some similar behind-the-scenes processing of the initial retrieved events that is causing the extra triggering.

Here is some documentation about how to throttle an alert to reduce the frequency of alert triggering:

http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/ThrottleAlerts

Hope this helps!

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 01:35:46 GMT

Thanks for the version info! Please see my comment below--I think throttling could help reduce the triggering.

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Tue, 12 Jan 2016 01:37:31 GMT

Ok, I've enabled throttling and will let you know soon if that works.

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 01:48:00 GMT

Also--to dig a bit more into how your query is returning events and to see if there is a way to modify it to establish more efficient triggering, could you post some info from the results that the query returns? For one event, could you post examples of what's in the following fields?

_raw
clientID
class

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 01:52:36 GMT

Yes, please do!

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Tue, 12 Jan 2016 19:09:29 GMT

The double alerts are gone. Instead I'm now seeing event alerts that shouldn't trigger because 'ClientId=some-id-3' and my query (see above) should filter those out. But that's a different problem.

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Tue, 29 Sep 2020 08:24:56 GMT

I'll give you the _raw data which contains all the fields of an event:
+++ 2016-01-12 17:57:28 +++
ClientId=my-clientid
HostId=my-hostid
Hostname=my-hostname
Platform=Linux (Ubuntu 12.04.5 LTS)
Versions=app1:1.3.3,app2:1.2.0
Mode=Installation
Class=PROCESS_NOT_RUNNING
Msg=One or more processes are not running

(I've replaced some field values since they contain sensitive data)

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 12 Jan 2016 21:25:45 GMT

Ok, sounds like throttling fixed the double triggering. Perhaps there's a way to rewrite the query to deal with the other aspect of the problem.

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 29 Sep 2020 08:25:04 GMT

Going back to your query:

index=tv- ClientId NOT (some-id-1 OR some-id-2 OR some-id-3) ClientId="" Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

If I'm interpreting the query correctly, then the events you want to alert on should follow these parameters
--ClientId== anything other than "some-id-1", "some-id-2", or "some-id-3"

--Class== not "success_first_attempt" or "Server did not accept key"

--Mode== only installation

If the above is correct, what happens if you remove this from your query: Client = "*"?

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Tue, 29 Sep 2020 08:25:07 GMT

--ClientId== anything other than "some-id-1", "some-id-2", or "some-id-3"
That's almost correct, see below.

--Class== not "success_first_attempt" or "Server did not accept key"
Class== not "success_first_attempt" is correct, the phrase "Server did not accept key" would appear in the 'Msg' key. So properly this part would be:
--Class== not "success_first_attempt" and --Msg== not contain "Server did not accept key"
but it also works if 'Msg' is omitted.

--Mode== only installation
That's correct

what happens if you remove this from your query: Client = "*"?
ClientId can be empty (ClientId=). In that case we don't want to trigger the alert. That's why I have the expression ClientId="*" in there.

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 29 Sep 2020 08:25:12 GMT

Give this alternative query a try:

index=tv- (ClientId=* AND ClientId NOT (some-id-1 OR some-id-2 OR some-id-3)) Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

Re: Triggered Alerts are Occuring twice on every Event that happens once

dschnabel — Wed, 13 Jan 2016 00:04:20 GMT

Ok, and what should I set for "Per result throttling fields"?

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Wed, 13 Jan 2016 00:23:55 GMT

I'm running a couple of questions about your query by my colleagues who work on the search language. You could try the query without throttling, to see if it yields the results you want. I'll report back if there's further advice we can offer! Stay tuned.

Re: Triggered Alerts are Occuring twice on every Event that happens once

frobinson_splun — Tue, 29 Sep 2020 08:25:15 GMT

Ok, my colleagues got back with a slightly different query suggestion. Try it without throttling to see if it yields the results you want.

index=tv- ClientId !="" ClientId NOT (some-id-1 OR some-id-2 OR some-id-3) Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

The difference is in using ClientId !="" vs. ClientId=. According to my colleagues, if you use ClientId=, you will get events that have this field, even if the value for that field is an empty string. Try !="" to avoid the events with empty ClientId values.