Splunk Search

Splunk SPL to detect anomaly over usages from indexes

bhilim
Loves-to-Learn Lots

Hello ,

I would really appreciate  your help in creating a splunk search query to find out the anomaly over size from individual indexes .There are 50+ indexes logging to splunk and I want some kind of alerting to notify if any of those index get sudden surge in logging from the normal trend.

 

 

 

Labels (2)
0 Karma

bhilim
Loves-to-Learn Lots

Thanks for your inputs @swong_splunk  .

I was trying somethings as below with outlier however it is not working as it should .

index="_internal" source="/opt/splunk/var/log/splunk/metrics.log" group=per_index_thruput series=** splunk_server=* earliest=-61m@m latest=-1m@m |rename series as index| eval GB=kb/(1024*1024) | bin _time span=1m | stats sum(GB) as size by _time index
| streamstats  avg("size") as avg stdev("size") as stdev by "index"
| eval lowerBound=(avg-stdev*exact(3)), upperBound=(avg+stdev*exact(3))
| eval isOutlier=if('size' < lowerBound OR 'size' > upperBound, 1, 0) |eval
time=strftime(_time, "%m/%d/%y %I:%M:%S:%p")| fields index time size
0 Karma

swong_splunk
Splunk Employee
Splunk Employee

In the License Usage Previous 30 days tab, there is a report Average and Peak Daily Volume that provides the avg volume and peak.  You can compute what a surge value would look like and alert on that value.

Add something like  | eval surge=avgVolumeGB*1.25

Or perform a calculation of peak compared to average and alert on that threshold.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...