At the moment I am running a search on a some log files, and looking to trigger an alert when the number of events has dropped by over 80%.
Currently, my alert triggers on a cron schedule every 5 minutes and the most recent time on the triggered alert, it always has a count of 0. With a drop of however many 5 minutes previously had been counted.
This is my search at present:
index=iis earliest=-20m "stringtomatchhere" | timechart span=5m count | delta count as difference | eval percentDifference =round(abs(difference/(count - difference))*100)
Where I am looking to find all events that match the string which forms part of the uri-stem, chart this at 5 min gaps for the past 20mins.
delta the count as difference, and then get the percentage difference.
An example of results returned :
The tigger conditions are search difference < 0 AND percentDifference > 80 - so I would like to show only where there is a drop in number of events, and that drop is of min 80%.
I have it set to run on a cron schedule every 5 mins with a 10m, -5m window.
If anyone could help point me in the right direction be much appreciated, just learning the ways of the splunking force.
Thanks for the help
When you run your searches every 5 minutes, splunk considers the current time also into timechart, ie; let's say you are running search at 10:30:01 and splunk considers 10:30-10:35 also (partial bucket) and all of your events might not have reached splunk by that time.
To avoid this , you can enable the option
partial=false in timechart and this will exclude any partial buckets (beginning or end).
You can also add a 0 as dummy value for the first bucket (
eval difference=coalesce(difference,0)) since there are no previous value to compare, not mandatory though.
The final search will be
index=iis earliest=-20m "stringtomatchhere" | timechart span=5m partial=false count | delta count as difference|eval difference=coalesce(difference,0) | eval percentDifference =round(abs(difference/(count - difference))*100)|where (difference < 1 AND percentDifference > 80)
Thanks very much, that is a great help,
I am still learning the abilities and workings of this fantastic splunk software.
Hi Steve - if the answer from @renjith.nair worked for you be sure to "accept" it . There should be a button above the comments section, next to "Award Points"