Alerting

Create an alert email from a splunk search result

ToKnowMore
Explorer

Hello,

I would like to create an alert based email on the following manually entered search string below.  The time frame used was for a 3 minute period, say from 1:02am to 1:05am, 10/2/2020.  As one can see from the result only has 3 reasons are present and that is fine as I know more would be reported if there were other reasons to be reported.

index=firewall host=10.10.10.10 | top limit=20 reason

Below is an example of the output:

reason                                count                      percentage
Idle Timeout                        582                      88.7197512
Transport Closing                42                           6.402439
DPD Failure                             32                           4.878049

It is my desire to have this alert be generated any time a reason is equal to or greater than 70% for a 3 minute period.  The trigger would be any reason passing that threshold percentage of 70%.  I understand this is considered "rolling window triggering" as such the following document was referred to me: 

docs.splunk.com/Documentation/SplunkCloud/latest/Alert/DefineRealTimeAlerts#Create_a_real-time_alert_with_rolling_window_triggering

That said, I did not find those instructions to be helpful for a percentage threshold trigger alert.  Perhaps what I am hoping to do cannot be done.  Nonetheless, I thought I would inquire with the Splunk community.

FYI, we are on code 7.3.5 and have no idea when an upgrade is taking place and to what code version.

Your time, help, patience and feedback is appreciated.

 

Labels (1)
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Time range (last 8 minutes) sets the outer limits for the query i.e. earliest 8 minutes ago, latest now. This is then restricted by the actual query (earliest="-7m@m" latest="-3m@m")

cron expression defines when the query is executed for the alert. The first part is the minute of the day, so */4 means every 4th minute. (*/2 would be every other minute, */15 would be every quarter of an hour). The rest of this expression means every hour, every day of the month, every month, every day of the week.

Expires is how long the alert remains on the list of activated alerts before it expires and is removed from the list of alerts. Essentially, you have 24 hours to look at the alert.

Trigger the alert when a custom search returns results, so in this case, when the query has a result with percent >= 75

View solution in original post

0 Karma

ToKnowMore
Explorer

Hi @ITWhisperer 

Sorry I missed that early suggestion, I apologize.

I have made the suggested change:

 

Last8mins.jpg

 

I suspect with "where count > 20" in play only those REASON codes hitting that metric of 20 would be listed in the email alert, no others is that correct?

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Correct, anything 20 or under will not appear in the final results (although they will have been included in the total for the relevant time slot, and therefore the percentages will still be correct).

0 Karma

ToKnowMore
Explorer

Hi @ITWhisperer 

Thanks for the prompt reply!

I apologize for not making the recommended "time range" change from 3 minutes to 8 minutes as I missed that suggestion.

Have updated accordingly and will advise results.

Last8mins.jpg

0 Karma

ToKnowMore
Explorer

Hi @ITWhisperer 

Hope this entry finds you well.

I have "normalized" the search string, or what I think is how the search string will be based on your input:

index=firewall host=10.10.10.10 earliest="-7m@m" latest="-3m@m" | bin span=2m _time | stats count by _time, reason | eventstats sum(count) as total by _time | eval percent=count * 100 / total | sort - percent | where count > 100 | head 20

The time range is set for last 8 mins and the "trigger alert when" is set for 75 (search percent>75).

I have not received any email alerts since the above in place yesterday and am unsurprised.  I suspect that minum count of 100 and the percent threshold of 75 has not been met. 

It is my hope that the true test will be next week when we've seen the event of interest take place.  I just did not want to discount any other time, hence the rolling trigger alert. 

If I may, I will provide an update on Tuesday, 10/13/2020.  I hope this is acceptable.

Thanks for your time, help and patience!

0 Karma

ToKnowMore
Explorer

Hi @ITWhisperer 

Hope this message finds you well.

The email alert worked.   I received an email alert in the early morning hours with the expected reason event identified.

Thank you!! 

For the sake of those who may find this configuration useful, could you please explain what the below search string means and the additional configuration components (e.g. the blue arrows) needed for a scheduled alert with a rolling window?

index=firewall host=10.10.10.10 earliest="-7m@m" latest="-3m@m" | bin span=2m _time | stats count by _time, reason | eventstats sum(count) as total by _time | eval percent=count * 100 / total | sort - percent | where count > 100 | head 20

 

Final Config alert.jpg

Example email alert received:

  Final email alert.jpg

It is my hope your explanation of the config would be better articulated from you.  Once done, I will gladly report your concise explanation as the solution.  For sure you have earned that.

Thanks again for your time, patience, and input throughout this topic!

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Time range (last 8 minutes) sets the outer limits for the query i.e. earliest 8 minutes ago, latest now. This is then restricted by the actual query (earliest="-7m@m" latest="-3m@m")

cron expression defines when the query is executed for the alert. The first part is the minute of the day, so */4 means every 4th minute. (*/2 would be every other minute, */15 would be every quarter of an hour). The rest of this expression means every hour, every day of the month, every month, every day of the week.

Expires is how long the alert remains on the list of activated alerts before it expires and is removed from the list of alerts. Essentially, you have 24 hours to look at the alert.

Trigger the alert when a custom search returns results, so in this case, when the query has a result with percent >= 75

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...