All Apps and Add-ons

Alert Throttling Whilst True

matthewcanty
Communicator

Hi, is it possible to throttle alerts whilst conditions are sustained.

For example:
Search result per minute = 10, 9, 12, 8, 2, 4, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 10, 10, 10, 8, 9, 3, 0 , 0 , 0, etc

So, I want my alert to be triggered at the zeros in bold, not for each subsequent one. So a service may stop running, but I don't want to receive constant reminders that it is not running, equally when I restart it again I want the alerts to become active again.

Tags (2)
0 Karma
1 Solution

Ledion_Bitincka
Splunk Employee
Splunk Employee

Yes you should be able to achieve that without even using throttling.

  1. run the search over two minutes
  2. compute the statistics that you need for previous minute and current minute
  3. only trigger the alert when latest == 0 and earlier != 0

One way to do this is by using this search

.... | timechart span=1m count | eventstats sum(count) as total | tail 1 | search count=0 total!=0

This search will return one result if the most recent minute has no events but the previous one has some.

View solution in original post

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

So here's the way to setup this alert based on "I have an int field called Fixtures in each event, and I want to know when it reaches zero, but not any subsequent zeros. So if(LastThreeMinutesHasFixtures && !LastTwoMinutesHasFixtures) ALERT"

  1. search over a 3 minute period
  2. compare last two minutes with first

    .... | stats last(Fixture) AS FixtureThreeMinutesAgo, sum(Fixture) AS Total | where Total>0 AND Total==FixtureThreeMinutesAgo

FixtureThreeMinutesAgo - is the value of Fixture 3 minutes ago, remember that Splunk processes events in descending time order

Total - the sum of all three minutes, so if Total==FixtureThreeMinutesAgo then the last two minutes have no Fixture

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Yes you should be able to achieve that without even using throttling.

  1. run the search over two minutes
  2. compute the statistics that you need for previous minute and current minute
  3. only trigger the alert when latest == 0 and earlier != 0

One way to do this is by using this search

.... | timechart span=1m count | eventstats sum(count) as total | tail 1 | search count=0 total!=0

This search will return one result if the most recent minute has no events but the previous one has some.

0 Karma

matthewcanty
Communicator

I'm kind of walking away from this for now, but in the future I'd like to get it figured. I have an int field called Fixtures in each event, and I want to know when it reaches zero, but not any subsequent zeros. So if(LastThreeMinutesHasFixtures && !LastTwoMinutesHasFixtures) ALERT

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

The reason it is not working is because stats will return only one result - you need two, current minute and previous minute. Can you please restate your problem but now using Fixtures and Markets?

0 Karma

matthewcanty
Communicator

I have to add two fields together first. Not sure what 'doing the right thing' will look like. My search now ... | stats sum(Fixtures) as f sum(Markets) as m | eval count=f+m | timechart span=1m count | eventstats sum(count) as total | tail 1 | search count=0 total!=0

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

You should set the alert mode to once per search and send you an alert when the number of results is greater than 0. Can you verify that the search including the tail cmd is doing the right thing?

0 Karma

matthewcanty
Communicator

Seems to be sending me constant alerts. How should the options be setup? Conditions/Alert Mode etc???

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...