Possibly to alert based on previous sample comparison

[Silah]

Hi

Put simply, I am trying to wrap my head around how I can configure an alert to trigger is a metric is X% higher or lower than the same metric, say 1 day ago.

So for example if I search

index=my_index eventStatus=fault | stats count by eventStatus

Searching "Last 15 minutes", giving say 100 results, can I trigger an alert IF the same search in the same 15 minute timeframe 1 day ago is for example 10% higher or lower?

Thanks

[ITWhisperer · Accepted solution]

Try something like this (note that if your time range spans midnight, then you will have to do something else with the bin _time)

index=my_index eventStatus=fault [| makeresults
    | eval row=mvrange(0,2) 
    | mvexpand row
    | addinfo
    | eval earliest=relative_time(info_min_time,(row*-1)."d")
    | eval latest=relative_time(info_max_time,(row*-1)."d")
    | table earliest latest]
| bin _time span=1d
| chart count by eventStatus _time
| foreach 1*
    [eval diff=if(isnull(diff),'<<FIELD>>',abs((diff-'<<FIELD>>')/diff))]
| where diff >0.15

[Silah]

Thanks, this seem to be producing something like what I am looking for.

Can I ask, what is the significance of this? I don't really understand it

'<<FIELD>>'

 Thanks

[ITWhisperer]

The foreach command goes through each field listed in the foreach command, in this instance, fieldnames beginning with 1 followed by anything. The time values are all epoch times, which are the number of seconds since the beginning of 1970. At present, these all start with 1. Eventually, in a about 9 years time, this will start with 2. So, within the subsearch of the foreach command (within the square brackets []), the <<FIELD>> value in the subsearch is replaced by the field name from the list. Since, in this case, this is a number, the <<FIELD>> is placed in single quotes '<<FIELD>>' to tell Splunk that it is to be interpreted as a field name (not a number).

[bowesmana]

If you search both time segments then work out which group the time belongs to, then compare the two

See this example

index=_audit (earliest=-1d@d latest=-1d@d+15m) OR (earliest=@d latest=@d+15m)
| eval group=if(_time>relative_time(now(),"@d"), "Prev", "Current")
| chart count over user by group
| eval alert=if(Current > Prev * 1.15, 1, 0)

So this sets group according to where _time sits then just chart over user and calculate excess

[Silah]

Thanks, I tried this but it only seems to list results that ocurred between 00:00 and 00:15 despite the search being "15 minutes ago"

[ITWhisperer]

That's why my solution uses addinfo which gives you the "earliest" and "latest" times from the timepicker