topic How to edit my timechart search to alert when the number of events has dropped by over 80%? in Splunk Search

How to edit my timechart search to alert when the number of events has dropped by over 80%?

insaneteddie — Thu, 11 Feb 2016 08:59:30 GMT

At the moment I am running a search on a some log files, and looking to trigger an alert when the number of events has dropped by over 80%.

Currently, my alert triggers on a cron schedule every 5 minutes and the most recent time on the triggered alert, it always has a count of 0. With a drop of however many 5 minutes previously had been counted.

This is my search at present:

index=iis   earliest=-20m  "stringtomatchhere" | timechart  span=5m count   | delta  count as difference   | eval  percentDifference =round(abs(difference/(count - difference))*100)

Where I am looking to find all events that match the string which forms part of the uri-stem, chart this at 5 min gaps for the past 20mins.
delta the count as difference, and then get the percentage difference.

An example of results returned :

The tigger conditions are search difference < 0 AND percentDifference > 80 - so I would like to show only where there is a drop in number of events, and that drop is of min 80%.

I have it set to run on a cron schedule every 5 mins with a 10m, -5m window.

If anyone could help point me in the right direction be much appreciated, just learning the ways of the splunking force.

Thanks for the help
S

Re: How to edit my timechart search to alert when the number of events has dropped by over 80%?

renjith_nair — Fri, 12 Feb 2016 03:49:56 GMT

When you run your searches every 5 minutes, splunk considers the current time also into timechart, ie; let's say you are running search at 10:30:01 and splunk considers 10:30-10:35 also (partial bucket) and all of your events might not have reached splunk by that time.

To avoid this , you can enable the option partial=false in timechart and this will exclude any partial buckets (beginning or end).

You can also add a 0 as dummy value for the first bucket (eval difference=coalesce(difference,0)) since there are no previous value to compare, not mandatory though.

The final search will be

index=iis   earliest=-20m  "stringtomatchhere" | timechart  span=5m partial=false count   | delta  count as difference|eval difference=coalesce(difference,0)   | eval  percentDifference =round(abs(difference/(count - difference))*100)|where (difference < 1 AND percentDifference > 80)

Re: How to edit my timechart search to alert when the number of events has dropped by over 80%?

insaneteddie — Fri, 12 Feb 2016 08:45:40 GMT

Thanks very much, that is a great help,
I am still learning the abilities and workings of this fantastic splunk software.
Steve

Re: How to edit my timechart search to alert when the number of events has dropped by over 80%?

jplumsdaine22 — Fri, 12 Feb 2016 10:55:49 GMT

Hi Steve - if the answer from @renjith.nair worked for you be sure to "accept" it . There should be a button above the comments section, next to "Award Points"