Splunk SPL to detect anomaly over usages from indexes

[bhilim]

Hello ,

I would really appreciate  your help in creating a splunk search query to find out the anomaly over size from individual indexes .There are 50+ indexes logging to splunk and I want some kind of alerting to notify if any of those index get sudden surge in logging from the normal trend.

[bhilim]

Thanks for your inputs @swong_splunk  .

I was trying somethings as below with outlier however it is not working as it should .

index="_internal" source="/opt/splunk/var/log/splunk/metrics.log" group=per_index_thruput series=** splunk_server=* earliest=-61m@m latest=-1m@m |rename series as index| eval GB=kb/(1024*1024) | bin _time span=1m | stats sum(GB) as size by _time index
| streamstats  avg("size") as avg stdev("size") as stdev by "index"
| eval lowerBound=(avg-stdev*exact(3)), upperBound=(avg+stdev*exact(3))
| eval isOutlier=if('size' < lowerBound OR 'size' > upperBound, 1, 0) |eval
time=strftime(_time, "%m/%d/%y %I:%M:%S:%p")| fields index time size

[swong_splunk]

In the License Usage Previous 30 days tab, there is a report Average and Peak Daily Volume that provides the avg volume and peak.  You can compute what a surge value would look like and alert on that value.

Add something like  | eval surge=avgVolumeGB*1.25

Or perform a calculation of peak compared to average and alert on that threshold.