How to setup alert to send an email when daily indexing volume limit is exceeded?

[kavraja]

Hey there,

I'm trying to set up a custom alert that would send out an email whenever the daily indexing volume is exceeded. The search I am running is:

index=_internal metrics kb group="per _index _thruput" series!= _*| eval totalGB = (kb / 1024) / 1024 | timechart span=1d sum(totalGB) as total

NB: I've put spaces between "per_index.." and so on so the formatting doesn't get confusing

Which shows how many gigs have been indexed for the day when run as the past 24 hours.

This works fine but my issue is that I can't figure out how to create the custom search that would monitor this search from the start of the day and send an alert when the gigs is above 2 for example.

In the alert option, I have scheduled it to run every hour and put in a custom search condition as "search total > 2". The problem with this is that the search only searches the indexing done for the past hour and not for the whole day. Meaning the results from the alert keep showing up as either 0.3 and so on every hour.

Is there a way I can run the alert every hour but have it take into account the amount indexed for the whole day?

Thanks in advance!

[martin_mueller · Accepted solution]

You can set the time range from @d to now and set a cron schedule of 1 * * * *, which would run the search every hour at one minute past.

Note, Splunk has a better way of calculating the current day's license usage - see Settings -> Licensing -> Usage Report, for example the top right panel "Today's Percentage of Daily License Quota Used per Pool" is basically what you need for your alert. That lists all your license pools along with a percentage used, set the alert to trigger if a pool reaches 90% or whatever you need.
To get the search behind the panel you can just click the magnifying glass in the bottom left corner of the panel, and save that as an alert.

[kavraja]

Thanks martin_mueller. It works great!