Splunk Search

How to set up alerts with multiple fields with different thresholds

Explorer

Trying to setup up an alert with multiple fields extracted through Field.

For example A,B, C etc and each having a different threshold for avg(time) and count.

Below is an example:

index=XXXX sourcetype="XXX" | where Field in("A") | stats count avg(time) as A | where A>2 and count>3| where Field in("B") |stats count avg(time) as B | where B>5 and count>10

Explorer

Able to get the data with above mentioned query but having trouble building alert since it's a custom alert.

Could you please send syntax for this
Condition 1: alert_a = 1 --> actions
Condition 2: alert_b = 1--> actions

0 Karma

Esteemed Legend

You cannot do it in the alert dialog so do it just like you are in SPL at the end and set your alert trigger to number of results and is greater than zero. This is more clear to the analysts anyway.

0 Karma

Splunk Employee
Splunk Employee

Splunk Investigate Alerts don't support Number of Results > foo

0 Karma

Esteemed Legend

WHAT???? Are you sure? That is CRAZY!

0 Karma

Splunk Employee
Splunk Employee

Yep. You'd append a | stats count() to your base search to approx that behavior. For now. Welcome any feedback -- scs-alerts@splunk.com and encourage you to sign up for the Investigate trial!

0 Karma

Builder

Can you write out some example data? I'm having a hard time coming up with a catch all answer for a few different cases I can think of that you might be talking about, and don't want to lead you down the wrong path!

0 Karma

Explorer

example in this case would be trigger an alert when avg(time) for A > 2 and count >3 then evaluate avg(time) for B >5 and count > 10. If all the conditions meets then only trigger an alert.

0 Karma

Splunk Employee
Splunk Employee

Have an answer pending post...

0 Karma

Splunk Employee
Splunk Employee

Hi there -

Currently we don't support multiple fields in separate the same condition (e.g. you can do A>0 and A<10 but not A>0 and B>0).

The work around is to create a new single column that represents the underlying logic of the column combination e.g.

From original
index=XXXX sourcetype="XXX"
| where Field in("A")
| stats count avg(time) as A
| where A>2 and count>3 --condition1
| where Field in("B")
|stats count avg(time) as B
| where B>5 and count>10 --condition2

Change base search to something along the lines of:
index=XXXX sourcetype="XXX"
| eval a_or_b=case(Field in("A"), "A", Field in("B"), "B")
| stats count() as myCount, avg(time) as avg_time by a_or_b
| eval alert_a=case(a_or_b="A" AND avg_time>2 AND count>3, 1)
| eval alert_b=case(a_or_b="B" AND avg_time>5 AND count>10, 1)

In the UI....
Condition 1: alert_a = 1 --> actions
Condition 2: alert_b = 1--> actions

Also please feel free to email scs-alerts@splunk.com if you run into any additional trouble!

0 Karma

Explorer

Able to get the data with above mentioned query but having trouble building alert since it's a custom alert.

Could you please send syntax for this
Condition 1: alert_a = 1 --> actions
Condition 2: alert_b = 1--> actions

0 Karma

Splunk Employee
Splunk Employee

Hi there - you don't need syntax for that. You simply designate the numeric field in the Splunk Investigate triggers & alerts UI and whether <>= a particular value.

See https://docs.splunk.com/Documentation/SplunkInvestigate/Current/Use/Trigger

0 Karma

Splunk Employee
Splunk Employee

Happy to help further w your specific case if you email scs-alerts@splunk.com

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!