topic setting up alert for search on index when counts for parameter are 0 in Splunk Search

setting up alert for search on index when counts for parameter are 0

jdepp — Thu, 05 Nov 2015 14:37:01 GMT

I am able to set up alerts for an index when the count = 0 for a specific parameter value. Since I have over 50 over these values, it is cumbersome to create so many alerts and wondered if I can create one alert that is able to monitor across these values. Let me explain with example:

source="/statsData.newIndex/tcp/10014"

This is the general search for the main index but for the specific parameter value:

source="/statsData.newIndex/tcp/10014" PStream=864

So I set up alerts for multiple values of this PStream field that monitors the count, and alerts when count =0 for the last hour. Is there a way to construct the search so that it monitors all values of this field PStream and for each one if the count =0, I will get the appropriate alert?

Re: setting up alert for search on index when counts for parameter are 0

jeffland — Thu, 05 Nov 2015 15:01:06 GMT

How about this:

source="/statsData.newIndex/tcp/10014" | stats count by PStream | where count=0

This search should give you a result with a line for each value of PStream with zero count. If you create an alert on that, you could either go easy and contain the search result in that one email (so the recipient has to look at it to determine which PStream(s) are affected).

Or, if you need to explicitly send an email for each PStream, you could create a lookup for the appropriate address for each PStream and use the sendemail command with a map command (docs here and here). I have to admit I'm not sure about that second method, it appears that it has been asked before, I only skimmed the answers though.

Re: setting up alert for search on index when counts for parameter are 0

woodcock — Thu, 05 Nov 2015 15:01:33 GMT

Like this:

... source="/statsData.newIndex/tcp/10014" | stats count(eval(PStream="864")) AS countPstream864 ... count(eval(otherField="otherValue")) AS countOtherValue | transpose| rename "row 1" AS count | where count=0

Then setup alert for Number of Events > 0.

Re: setting up alert for search on index when counts for parameter are 0

woodcock — Thu, 05 Nov 2015 15:02:46 GMT

The problem is that you will only be counting things that have non-zero instances (and he needs the opposite).

Re: setting up alert for search on index when counts for parameter are 0

jeffland — Thu, 05 Nov 2015 15:07:05 GMT

Oh. You're right. Didn't think this through apparently.
In that case, ignore the first part of this answer - but the second one might still be useful if you intend to send emails depending on which PStream is affected.

Re: setting up alert for search on index when counts for parameter are 0

jdepp — Thu, 05 Nov 2015 15:33:06 GMT

thanks for your suggestion.

Re: setting up alert for search on index when counts for parameter are 0

jdepp — Thu, 05 Nov 2015 15:35:24 GMT

Thanks for your help. Does this search apply to all where PSTream=* or do I add for each PStream='864', '865', '866'

countPstream864 ... count

why are there dots in between?

Re: setting up alert for search on index when counts for parameter are 0

woodcock — Thu, 05 Nov 2015 15:39:28 GMT

The dots are showing you where you need to add your "over 50 of these" (as you put it in your question). So for the 3 values you just noted, it would look like this:

... source="/statsData.newIndex/tcp/10014" | stats count(eval(PStream="864")) AS countPstream864 count(eval(PStream="865")) AS countPstream865 count(eval(PStream="866")) AS countPstream866 | transpose| rename "row 1" AS count | where count=0