Alerting
Highlighted

Index volume: hourly increase and raw volume

Influencer

So far I've been dissatisfied with the various volume-used searches I've tried. My latest attempt includes not only the volume in the last full hour, but also the % change from the previous hour, as well as the busiest source host and its volume.

index=_internal group="per_index_thruput" NOT series="_*" NOT series="history" NOT series="summary" earliest=-2h@h latest=@h | eval mb=kb/1024| timechart span=1h sum(mb) by series | delta main as hourly_change | eval perc_change=hourly_change/(main-hourly_change) | fields + perc_change,main | tail 1 | appendcols [search earliest=-1h@h latest=@h | eval KB=length(_raw)/1024 | stats sum(eval(KB/1024)) as MB by host | sort -MB | head 1 ]

Then I alert on a where clause for total MB and/or percentage change hour-over-hour in an attempt to catch big jumps in volume or rates that, if sustained, would push our licensed limit.

What's your preferred method of tracking/alerting on volume?

0 Karma
Highlighted

Re: Index volume: hourly increase and raw volume

Esteemed Legend

You should probably look at Timewrap which will allow you to much more easily compare and correlate overlapping time segments:

https://splunkbase.splunk.com/app/1645/

0 Karma