- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to monitor drop in events per index. What is the best way to get a baseline and detect deviation to the volume? I am more interesting in drop in events and not increase.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Start with per_index_thruput in _internal. It's just a sample and has natural ups and downs, but may give you something to work with.
index=_internal sourcetype=splunkd source=*metrics.log* group=per_index_thruput
If this reply helps you, Karma would be appreciated.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What I am looking for might be something even simpler. If I can get the total log volume per day and set up a threshold for alerting that will work. I was thinking log volume for most indexes (log sources) do tend to drop on the weekends. Perhaps there is a threshold that can be set up based on the day of the week. Weekends vs week days. Any such way to accomplish this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Check the Monitoring Console to see if it has a query that comes close to what you want.
As for accounting for weekends and holidays, you probably should look at the Machine Learning Toolkit (MLTK). It has algorithms that can detect trends in your data and help find when the trends break.
If this reply helps you, Karma would be appreciated.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Start with per_index_thruput in _internal. It's just a sample and has natural ups and downs, but may give you something to work with.
index=_internal sourcetype=splunkd source=*metrics.log* group=per_index_thruput
If this reply helps you, Karma would be appreciated.
