topic Re: Possibly to alert based on previous sample comparison in Splunk Search

Possibly to alert based on previous sample comparison

Silah — Tue, 02 Jul 2024 10:48:32 GMT

Put simply, I am trying to wrap my head around how I can configure an alert to trigger is a metric is X% higher or lower than the same metric, say 1 day ago.

So for example if I search

index=my_index eventStatus=fault | stats count by eventStatus

Searching "Last 15 minutes", giving say 100 results, can I trigger an alert IF the same search in the same 15 minute timeframe 1 day ago is for example 10% higher or lower?

Thanks

Re: Possibly to alert based on previous sample comparison

bowesmana — Tue, 02 Jul 2024 11:53:02 GMT

If you search both time segments then work out which group the time belongs to, then compare the two

See this example

index=_audit (earliest=-1d@d latest=-1d@d+15m) OR (earliest=@d latest=@d+15m) | eval group=if(_time>relative_time(now(),"@d"), "Prev", "Current") | chart count over user by group | eval alert=if(Current > Prev * 1.15, 1, 0)

So this sets group according to where _time sits then just chart over user and calculate excess

Re: Possibly to alert based on previous sample comparison

ITWhisperer — Tue, 02 Jul 2024 12:19:56 GMT

Try something like this (note that if your time range spans midnight, then you will have to do something else with the bin _time)

Re: Possibly to alert based on previous sample comparison

Silah — Tue, 02 Jul 2024 13:57:55 GMT

Thanks, I tried this but it only seems to list results that ocurred between 00:00 and 00:15 despite the search being "15 minutes ago"

Re: Possibly to alert based on previous sample comparison

Silah — Tue, 02 Jul 2024 14:25:19 GMT

Thanks, this seem to be producing something like what I am looking for.

Can I ask, what is the significance of this? I don't really understand it

'<<FIELD>>'

Thanks

Re: Possibly to alert based on previous sample comparison

ITWhisperer — Tue, 02 Jul 2024 14:35:20 GMT

That's why my solution uses addinfo which gives you the "earliest" and "latest" times from the timepicker

Re: Possibly to alert based on previous sample comparison

ITWhisperer — Tue, 02 Jul 2024 14:43:32 GMT

The foreach command goes through each field listed in the foreach command, in this instance, fieldnames beginning with 1 followed by anything. The time values are all epoch times, which are the number of seconds since the beginning of 1970. At present, these all start with 1. Eventually, in a about 9 years time, this will start with 2. So, within the subsearch of the foreach command (within the square brackets []), the <<FIELD>> value in the subsearch is replaced by the field name from the list. Since, in this case, this is a number, the <<FIELD>> is placed in single quotes '<<FIELD>>' to tell Splunk that it is to be interpreted as a field name (not a number).