How to monitor deviation to log volume?

[Splunk77]

I am trying to monitor drop in events per index. What is the best way to get a baseline and detect deviation to the volume? I am more interesting in drop in events and not increase.

[Splunk77]

What I am looking for might be something even simpler. If I can get the total log volume per day and set up a threshold for alerting that will work. I was thinking log volume for most indexes (log sources) do tend to drop on the weekends. Perhaps there is a threshold that can be set up based on the day of the week. Weekends vs week days. Any such way to accomplish this?

[richgalloway]

Check the Monitoring Console to see if it has a query that comes close to what you want.

As for accounting for weekends and holidays, you probably should look at the Machine Learning Toolkit (MLTK).  It has algorithms that can detect trends in your data and help find when the trends break.

[richgalloway · Accepted solution]

Start with per_index_thruput in _internal.  It's just a sample and has natural ups and downs, but may give you something to work with.

index=_internal sourcetype=splunkd source=*metrics.log* group=per_index_thruput