Splunk Search

Manage alert threshold and rows returned all within the custom condition in the alert

Cuyose
Builder

This kind of spiraled as I was helping a coworker with an alert they had all the duration and times hardcoded in the query of the alert.

The easiest way to describe what I am trying to do would be if Splunk allowed you to configure an alert for

where value>X and rows returned > y

So an alert for the sum of action durations by time could be managed without specifying the duration threshold or time within the query but with the custom trigger condition within the alert.

Has anyone run into this and figured it out?

Basically I am now trying to work in a single stanza at the end of a query (which I can use as the custom condition in the alert) to evaluate the rows returned based on the value per row in the result set

0 Karma
1 Solution

Cuyose
Builder

Ok I think I figured this out. I end the base search in the alert like this(its pretty verbose in the field names to help me remember things)

| bin span=1m _time
| stats exactperc95(DurationTotal) AS p95Duration by date_minute Action 
| outlier
| addtotals labelfield=Action label=Total 
| stats sum(Total) as DurationTotal dc(date_minute) as consecutive_minutes_in_violation by date_minute 

and the custom condition in the alert

| where DurationTotal>270000 | streamstats sum(consecutive_minutes_in_violation) AS consecutive_minutes_in_violation| search consecutive_minutes_in_violation>1

This allows not only the Duration threshold to be edited but also the consecutive minutes it crossed the threshold independent of the search time frame. Nothing therefor is hard coded in the actual alert query.

View solution in original post

0 Karma

Cuyose
Builder

Ok I think I figured this out. I end the base search in the alert like this(its pretty verbose in the field names to help me remember things)

| bin span=1m _time
| stats exactperc95(DurationTotal) AS p95Duration by date_minute Action 
| outlier
| addtotals labelfield=Action label=Total 
| stats sum(Total) as DurationTotal dc(date_minute) as consecutive_minutes_in_violation by date_minute 

and the custom condition in the alert

| where DurationTotal>270000 | streamstats sum(consecutive_minutes_in_violation) AS consecutive_minutes_in_violation| search consecutive_minutes_in_violation>1

This allows not only the Duration threshold to be edited but also the consecutive minutes it crossed the threshold independent of the search time frame. Nothing therefor is hard coded in the actual alert query.

0 Karma

adonio
Ultra Champion

i might be completely off here and misunderstanding your question but if you have 2 conditions you would like to set an alert by, maybe use a sub-search? here is a very general example:

your first search | appendcols [search that's finds stuff / matches condition | eval condition=True| return condition] | search condition=True

If number of results >= 1, trigger alert
credit for the approach to @jkat54
hope it helps

0 Karma

Cuyose
Builder

I think you understood, but I wanted to not have any of the conditions within the actual search itself, only the alert condition. I got it working though with the solution below.

0 Karma

adonio
Ultra Champion

very good then.
kindly mark your answer as answered.
well done

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...