I would really appreciate your help in creating a splunk search query to find out the anomaly over size from individual indexes .There are 50+ indexes logging to splunk and I want some kind of alerting to notify if any of those index get sudden surge in logging from the normal trend.
I was trying somethings as below with outlier however it is not working as it should .
index="_internal" source="/opt/splunk/var/log/splunk/metrics.log" group=per_index_thruput series=** splunk_server=* earliest=-61m@m latest=-1m@m |rename series as index| eval GB=kb/(1024*1024) | bin _time span=1m | stats sum(GB) as size by _time index
| streamstats avg("size") as avg stdev("size") as stdev by "index"
| eval lowerBound=(avg-stdev*exact(3)), upperBound=(avg+stdev*exact(3))
| eval isOutlier=if('size' < lowerBound OR 'size' > upperBound, 1, 0) |eval
time=strftime(_time, "%m/%d/%y %I:%M:%S:%p")| fields index time size
In the License Usage Previous 30 days tab, there is a report Average and Peak Daily Volume that provides the avg volume and peak. You can compute what a surge value would look like and alert on that value.
Add something like | eval surge=avgVolumeGB*1.25
Or perform a calculation of peak compared to average and alert on that threshold.