topic Re: finding greater than value from search resutls in Splunk Search

finding greater than value from search resutls

gerald_huddlest — Thu, 09 Feb 2012 10:37:56 GMT

hi
I have created an eventtype that looks for a certain event across 12 servers (cmchost). I created a dashboard showing showing the number of events per cmchost over time.

I would like to set a threshold to alert when the number of events per server in a 15 minute period is exceeded but am struggling to put this last part together.

Dashboard search is:
tag=failure | dedup _raw | timechart count by CmcHost

I want an alert when a cmchost exceeds 10 events in 15 minute period.

grateful for some advise

Re: finding greater than value from search resutls

Drainy — Thu, 09 Feb 2012 15:42:26 GMT

have a look at;
http://docs.splunk.com/Documentation/Splunk/4.3/User/SchedulingSavedSearches

Basically you don't need to do the count for it, just do the search to return the events. Through the scheduled search/alert screen set it to alert when the numnber of results exceeds 10 and schedule it to run over a 15 minute period or perhaps every 5 minutes for some overlap.

Re: finding greater than value from search resutls

gerald_huddlest — Fri, 10 Feb 2012 08:21:24 GMT

thanks for the resposne, but that will look for the events generated by all 12 servers. I only want to alert if 1 server exceeds 10 events in a 15 minute period which i why i was putting hte results of the original search into a table so that I can see how many events per server.
I dont want to run a seperate search for each server which would be the easy way to do it, but want to combine into a single search.

Re: finding greater than value from search resutls

eelisio2 — Fri, 10 Feb 2012 16:41:43 GMT

This will return only the servers with more than 10 events:

tag=failure | dedup _raw | stats count by CmcHost | search count > 10

This will only return rows where the count is greater than 10.
Then, you can alert if number of events(rows returned by the search) is greater than zero.

Re: finding greater than value from search resutls

gerald_huddlest — Fri, 10 Feb 2012 16:47:29 GMT

thanks, exactly what i was looking for!