How to display an alert if a field value pre defined threshold is breached

[nickfernand]

I an new to Splunk and amdeveloping it to provide monitoring of message flow across our landscape and would like to know how i can set a threshold, apply it to a field and then alert out on a dashboard if that threshold is breached.

Use case is a a message is flowing across between applications, if a transient step in between has a build up of messages in it that breach a pre defined threshold then display an alert on a dashboard which is monitored by our operatiosn group.

Thanks in advance

Nick

[martin_mueller]

There's two things potentially mixed up here: Displaying a breach of threshold on a dashboard, and triggering an alert based on a breach of threshold. From a search language point of view both are basically the same thing, but where you use them within Splunk isn't.

For the dashboard way, you could have a search like this:

search for your events with queue sizes | stats max(queue_size) as max_queue_size | rangemap field=max_queue_size low=0-9 elevated=10-19 default=severe

Use that search in a single value panel like this:

<single>
  <search>
    <query>see above</query>
  </search>
  <option name="classField">range</option>
</single>

That'll display the maximum queue size in a colour determined by the rangemap - there are classes pre-defined in CSS to magically do this.

The alert is a bit different, you save a search like this as an alert:

search for your events with queue sizes | stats max(queue_size) as max_queue_size

Then set it to execute on some schedule with some matching time range, for example to run every hour with -61m@m to -m@m as time range to allow for up to one minute of delay, and define the alert's trigger condition as where max_queue_size > 19 if it should alert above 19.