topic Re: Confused with Trigger Conditions - Confused by Documentation in Splunk Search

Confused with Trigger Conditions - Confused by Documentation

rogue_carrot — Tue, 29 Sep 2020 20:02:52 GMT

I am reading the documentation at the following page: http://docs.splunk.com/Documentation/Splunk/7.1.1/Alert/AlertTriggerConditions

The parts that do not make sense to me are, "Using a search with custom trigger condition" and then the next section of the webpage, "Using a search without a trigger condition". Both of these searches look the same. What is the difference between these two searches? Could someone please point out what is the trigger condition?

Here are the searches. The first one is the search with the custom trigger condition: index=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level

The next search is the one without the custom trigger condition:
log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level | search count > 10

Re: Confused with Trigger Conditions - Confused by Documentation

DalJeanis — Tue, 29 Sep 2020 20:02:57 GMT

Look at the very last paragraph of each section.

THE FIRST ONE

In this scenario, the original search
results detail the count for all log
levels, but the alert triggers only
when the log_level counts are greater
than ten. This means that all
log_level counts are available to use
as part of an alert notification.

THE SECOND ONE

In this case, the search results
include only log_level values that
are greater than ten. By comparison,
using a search with conditional
triggering in the previous example
means that results include counts for
all log level fields.

To reiterate:

In the first version, the search includes all results, regardless of the count. If there is any single result that triggers the alert, then the alert will include all results.

In the second version, only the particular results that trigger the alert will be sent.

Re: Confused with Trigger Conditions - Confused by Documentation

rogue_carrot — Wed, 20 Jun 2018 23:19:20 GMT

Thank-you for the reply DalJeanis. Could you please explicitly point out the trigger condition also?

Re: Confused with Trigger Conditions - Confused by Documentation

woodcock — Sat, 30 Jun 2018 23:21:22 GMT

Forgive me for not answering your question but let me give you some advice that will eliminate the need for an answer. It is a best practice to never use the savedsearches.conf threshold settings. The threshold conditions should always be contained in the search SPL. Here is why. The people who will be handling the search alert will first receive the email and you need for them to be able to see what the threshold actually was. If this is the last part of the search SPL, they will be able to see it. Some of these people will not even have access to Splunk to go to the search definition and most that do will not understand that they need to. So KISS and put the "real" threshold into the search SPL and then always use a savedsearches.conf trigger setting of if number of events > 0.