Sending conditional alerts based on previous search result

[technie101]

Following is the json log format being stored in Splunk.

{   data:[
     {
       "endpoint":"ep_1",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_2",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_3",
       "service":"service_2",
       "http_status_code":"503"
     }   ] }

Currently, an alert is setup to check once every 5 mins with the following search query and send an email if results>0.

host=something "data{}.http_status_code"= 5*| eval endpoint='data{}.endpoint' | Stats Count("data{}.http_status_code") as Count by endpoint | where Error_Count > 0

The generated result is :

Now, the ask is, once an alert is generated (which contains a particular endpoin, say ep_1), in the next run, if the search result contains the same endpoint, it should NOT send an email for another hour.

So in this case, when the search query runs after 5 mins and contains ep_1 in the result, it should not send an email. So effectively, I need a condition for this endpoint's occurrence in the previously executed search result.

Is there a way this can be accomplished? Please help.

[s2_splunk · Accepted solution]

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

[technie101]

Thanks. Trying this approach with a KVStore now.

For the above usecase, how do we compare the endpoint from the search results with the endpoint from the KVStore?

[inventsekar]

You can try the alert throttling option -

https://www.splunk.com/blog/2010/06/01/alert-throttling.html

http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Alert/ThrottleAlerts