Splunk Search

Splunk SPL to detect anomaly over usages from indexes

bhilim
Loves-to-Learn Lots

Hello ,

I would really appreciate  your help in creating a splunk search query to find out the anomaly over size from individual indexes .There are 50+ indexes logging to splunk and I want some kind of alerting to notify if any of those index get sudden surge in logging from the normal trend.

 

 

 

Labels (2)
0 Karma

bhilim
Loves-to-Learn Lots

Thanks for your inputs @swong_splunk  .

I was trying somethings as below with outlier however it is not working as it should .

index="_internal" source="/opt/splunk/var/log/splunk/metrics.log" group=per_index_thruput series=** splunk_server=* earliest=-61m@m latest=-1m@m |rename series as index| eval GB=kb/(1024*1024) | bin _time span=1m | stats sum(GB) as size by _time index
| streamstats  avg("size") as avg stdev("size") as stdev by "index"
| eval lowerBound=(avg-stdev*exact(3)), upperBound=(avg+stdev*exact(3))
| eval isOutlier=if('size' < lowerBound OR 'size' > upperBound, 1, 0) |eval
time=strftime(_time, "%m/%d/%y %I:%M:%S:%p")| fields index time size
0 Karma

swong_splunk
Splunk Employee
Splunk Employee

In the License Usage Previous 30 days tab, there is a report Average and Peak Daily Volume that provides the avg volume and peak.  You can compute what a surge value would look like and alert on that value.

Add something like  | eval surge=avgVolumeGB*1.25

Or perform a calculation of peak compared to average and alert on that threshold.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.