Splunk indexing volume by index alert

[splunkiscool1]

Hello,

I would like to get an alert if the indexing volume for an index drops.

I'm thinking something similar this, but I'm not sure how to translate it to Splunk's language:

if (yesterday's total indexing volume for EACH index) is at least 20% less than (the average daily indexing volume for EACH index over -2 to -7 days) then (alert)

Any suggestions?

[martin_mueller · Accepted solution]

Starting with the index volume query from the Status menu in Search you can build this:

index=_internal source=*metrics.log group=per_index_thruput | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume < avg_volume * 0.8

Run over -7d@d to @d and alert whenever this yields a result.

[martin_mueller]

Yesterday was Sunday, the where is hence looking for a day that was filtered out in the main search.

[splunkiscool1]

index=_internal source=*metrics.log group=per_index_thruput (date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday) | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume > avg_volume * 0.8

I get no results whether I have a < or >. When removing the (date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday) I do get results with the original search from your first post

[martin_mueller]

Append this to the main search:

(date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday)

[splunkiscool1]

Thanks martin!

I'm trying to figure out how to specify weekdays only(as this will trigger every weekend how it is now, because I have an expected large drop in volume during the weekend).

So, I would like to just ignore weekends, and only run the search throughout the week, but the average should only be calculated with weekdays as well.

I see in Splunk Documentation how to specify a specific day of the week, but how could I specify to go 7 days back, but only average the weekdays?