topic Re: Custom alert trigger count range in Splunk Enterprise

Custom alert trigger count range

jwv — Thu, 05 Dec 2024 20:22:59 GMT

I want my alert to trigger when the result count is between 250 and 500, trying to use the custom trigger condition in the alert setup with

search count => 250 AND search count <=500

but this is not working as expected. Even trying to to use the custom trigger condition for one condition like search count => 250 is not working. What is the right way to do this?

Re: Custom alert trigger count range

richgalloway — Thu, 05 Dec 2024 20:39:59 GMT

I've found the custom triggers to be unreliable at best. What works better is to put the alert condition in the search query and have the alert trigger when the number of results is not zero.

Re: Custom alert trigger count range

jwv — Mon, 09 Dec 2024 16:50:19 GMT

Thank you for the tip! Do you have any suggestions on how to format this query? I'm not sure the best way to do this when I need the alert to fire based of number of results.

Re: Custom alert trigger count range

richgalloway — Mon, 09 Dec 2024 17:04:32 GMT

Move the trigger condition from the alert to the search. IOW, put this on the end of the query

| where count => 250 AND count <=500

Re: Custom alert trigger count range

jwv — Wed, 11 Dec 2024 19:41:35 GMT

I've tried this a few ways with no success, I tried to just append the where condition on the end of my query so it looks something like this

index="my-index" | search "http 404" | stats count | where count => 250 AND count <= 100

but this still just returns the number of matching events and I run into the same problem trying to set up the alert.

I have also tried with eval to output a result like this

index="my-index" | search "http 404" | stats count | eval result=if(count>=250 AND count<=500, 1, 0) | table result

which properly returns 1 or 0 depending on if the number of results are in the range I am looking for, however it is still not alerting properly when the trigger is set to >1. I think the trigger is still running against the number of events which is also being returned (and I like that I can see this) and not the result I set up.

Any other suggestions would be much appreciated.

Re: Custom alert trigger count range

richgalloway — Wed, 11 Dec 2024 20:08:43 GMT

@jwv wrote:
I've tried this a few ways with no success, I tried to just append the where condition on the end of my query so it looks something like this

index="my-index" | search "http 404" | stats count | where count => 250 AND count <= 100

but this still just returns the number of matching events and I run into the same problem trying to set up the alert.

That the search "just returns the number of matching events" is to be expected since that is all it is told to do. The where command should cause the search to return a result only if that result is a number between 250 and 500; otherwise it should say "No results found". That is what will trigger the alert - if there is a result (Number of events > 0) it's because the search criteria were met.