Knowledge Management

Sending conditional alerts based on previous search result

technie101
Explorer

Following is the json log format being stored in Splunk.

{   data:[
     {
       "endpoint":"ep_1",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_2",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_3",
       "service":"service_2",
       "http_status_code":"503"
     }   ] }

Currently, an alert is setup to check once every 5 mins with the following search query and send an email if results>0.

host=something "data{}.http_status_code"= 5*| eval endpoint='data{}.endpoint' | Stats Count("data{}.http_status_code") as Count by endpoint | where Error_Count > 0

The generated result is :

alt text

Now, the ask is, once an alert is generated (which contains a particular endpoin, say ep_1), in the next run, if the search result contains the same endpoint, it should NOT send an email for another hour.

So in this case, when the search query runs after 5 mins and contains ep_1 in the result, it should not send an email. So effectively, I need a condition for this endpoint's occurrence in the previously executed search result.

Is there a way this can be accomplished? Please help.

0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

View solution in original post

s2_splunk
Splunk Employee
Splunk Employee

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

technie101
Explorer

Thanks. Trying this approach with a KVStore now.

For the above usecase, how do we compare the endpoint from the search results with the endpoint from the KVStore?

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...