Knowledge Management

Sending conditional alerts based on previous search result

Explorer

Following is the json log format being stored in Splunk.

{   data:[
     {
       "endpoint":"ep_1",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_2",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_3",
       "service":"service_2",
       "http_status_code":"503"
     }   ] }

Currently, an alert is setup to check once every 5 mins with the following search query and send an email if results>0.

host=something "data{}.http_status_code"= 5*| eval endpoint='data{}.endpoint' | Stats Count("data{}.http_status_code") as Count by endpoint | where Error_Count > 0

The generated result is :

alt text

Now, the ask is, once an alert is generated (which contains a particular endpoin, say ep_1), in the next run, if the search result contains the same endpoint, it should NOT send an email for another hour.

So in this case, when the search query runs after 5 mins and contains ep_1 in the result, it should not send an email. So effectively, I need a condition for this endpoint's occurrence in the previously executed search result.

Is there a way this can be accomplished? Please help.

0 Karma
1 Solution

Splunk Employee
Splunk Employee

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

View solution in original post

Splunk Employee
Splunk Employee

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

View solution in original post

Explorer

Thanks. Trying this approach with a KVStore now.

For the above usecase, how do we compare the endpoint from the search results with the endpoint from the KVStore?

0 Karma

Champion
0 Karma