All Apps and Add-ons

How to display an alert if a field value pre defined threshold is breached

nickfernand
New Member

I an new to Splunk and amdeveloping it to provide monitoring of message flow across our landscape and would like to know how i can set a threshold, apply it to a field and then alert out on a dashboard if that threshold is breached.

Use case is a a message is flowing across between applications, if a transient step in between has a build up of messages in it that breach a pre defined threshold then display an alert on a dashboard which is monitored by our operatiosn group.

Thanks in advance

Nick

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

There's two things potentially mixed up here: Displaying a breach of threshold on a dashboard, and triggering an alert based on a breach of threshold. From a search language point of view both are basically the same thing, but where you use them within Splunk isn't.

For the dashboard way, you could have a search like this:

search for your events with queue sizes | stats max(queue_size) as max_queue_size | rangemap field=max_queue_size low=0-9 elevated=10-19 default=severe

Use that search in a single value panel like this:

<single>
  <search>
    <query>see above</query>
  </search>
  <option name="classField">range</option>
</single>

That'll display the maximum queue size in a colour determined by the rangemap - there are classes pre-defined in CSS to magically do this.

The alert is a bit different, you save a search like this as an alert:

search for your events with queue sizes | stats max(queue_size) as max_queue_size

Then set it to execute on some schedule with some matching time range, for example to run every hour with -61m@m to -m@m as time range to allow for up to one minute of delay, and define the alert's trigger condition as where max_queue_size > 19 if it should alert above 19.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...