Splunk Search

How to edit my timechart search to alert when the number of events has dropped by over 80%?

insaneteddie
Path Finder

HI

At the moment I am running a search on a some log files, and looking to trigger an alert when the number of events has dropped by over 80%.

Currently, my alert triggers on a cron schedule every 5 minutes and the most recent time on the triggered alert, it always has a count of 0. With a drop of however many 5 minutes previously had been counted.

This is my search at present:

index=iis   earliest=-20m  "stringtomatchhere" | timechart  span=5m count   | delta  count as difference   | eval  percentDifference =round(abs(difference/(count - difference))*100)

Where I am looking to find all events that match the string which forms part of the uri-stem, chart this at 5 min gaps for the past 20mins.
delta the count as difference, and then get the percentage difference.

An example of results returned :
searchRes
The tigger conditions are search difference < 0 AND percentDifference > 80 - so I would like to show only where there is a drop in number of events, and that drop is of min 80%.

I have it set to run on a cron schedule every 5 mins with a 10m, -5m window.

If anyone could help point me in the right direction be much appreciated, just learning the ways of the splunking force.

Thanks for the help
S

0 Karma
1 Solution

renjith_nair
SplunkTrust
SplunkTrust

When you run your searches every 5 minutes, splunk considers the current time also into timechart, ie; let's say you are running search at 10:30:01 and splunk considers 10:30-10:35 also (partial bucket) and all of your events might not have reached splunk by that time.

To avoid this , you can enable the option partial=false in timechart and this will exclude any partial buckets (beginning or end).

You can also add a 0 as dummy value for the first bucket (eval difference=coalesce(difference,0)) since there are no previous value to compare, not mandatory though.

The final search will be

index=iis   earliest=-20m  "stringtomatchhere" | timechart  span=5m partial=false count   | delta  count as difference|eval difference=coalesce(difference,0)   | eval  percentDifference =round(abs(difference/(count - difference))*100)|where (difference < 1 AND percentDifference > 80)
Happy Splunking!

View solution in original post

renjith_nair
SplunkTrust
SplunkTrust

When you run your searches every 5 minutes, splunk considers the current time also into timechart, ie; let's say you are running search at 10:30:01 and splunk considers 10:30-10:35 also (partial bucket) and all of your events might not have reached splunk by that time.

To avoid this , you can enable the option partial=false in timechart and this will exclude any partial buckets (beginning or end).

You can also add a 0 as dummy value for the first bucket (eval difference=coalesce(difference,0)) since there are no previous value to compare, not mandatory though.

The final search will be

index=iis   earliest=-20m  "stringtomatchhere" | timechart  span=5m partial=false count   | delta  count as difference|eval difference=coalesce(difference,0)   | eval  percentDifference =round(abs(difference/(count - difference))*100)|where (difference < 1 AND percentDifference > 80)
Happy Splunking!

insaneteddie
Path Finder

Thanks very much, that is a great help,
I am still learning the abilities and workings of this fantastic splunk software.
Steve

0 Karma

jplumsdaine22
Influencer

Hi Steve - if the answer from @renjith.nair worked for you be sure to "accept" it . There should be a button above the comments section, next to "Award Points"

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...