Splunk Search

Splunk SPL to detect anomaly over usages from indexes

Loves-to-Learn Lots

Hello ,

I would really appreciate  your help in creating a splunk search query to find out the anomaly over size from individual indexes .There are 50+ indexes logging to splunk and I want some kind of alerting to notify if any of those index get sudden surge in logging from the normal trend.




Labels (2)
0 Karma

Loves-to-Learn Lots

Thanks for your inputs @swong_splunk  .

I was trying somethings as below with outlier however it is not working as it should .

index="_internal" source="/opt/splunk/var/log/splunk/metrics.log" group=per_index_thruput series=** splunk_server=* earliest=-61m@m latest=-1m@m |rename series as index| eval GB=kb/(1024*1024) | bin _time span=1m | stats sum(GB) as size by _time index
| streamstats  avg("size") as avg stdev("size") as stdev by "index"
| eval lowerBound=(avg-stdev*exact(3)), upperBound=(avg+stdev*exact(3))
| eval isOutlier=if('size' < lowerBound OR 'size' > upperBound, 1, 0) |eval
time=strftime(_time, "%m/%d/%y %I:%M:%S:%p")| fields index time size
0 Karma

Splunk Employee
Splunk Employee

In the License Usage Previous 30 days tab, there is a report Average and Peak Daily Volume that provides the avg volume and peak.  You can compute what a surge value would look like and alert on that value.

Add something like  | eval surge=avgVolumeGB*1.25

Or perform a calculation of peak compared to average and alert on that threshold.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!


Or Learn More in Our Blog >>