Alerting

Alert on "Number of results"

marthin
Engager
Labels (1)
Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @marthin,

let me understand: you configured the alert as the screenshot and you have 33 emails instead 1, is it correct?

At first never use real time alerts because it consumes too much resources (every search takes one CPU and release it when finished, with real time it doesn't release CPU!).

Then configure Throttle: Throttle is a run exclusion when an alert is fired for a defined time, to avoid thet the alert is triggered many times with the same condition.

Then, using real time your condition is continously verified and the aler triggers.

The best approach is to schedule your alers e.g. every 5 minutes or every hour, frequency usually depends on the time period of a search: e.g. if I have a time period of 1 hours my alert runs every hour, it isn't correct to have an alert with a time period of 24 hours that runs continously because you have many fired alerts!

So I hint to re-design your alert:

  • reducing time period,
  • setting a frequency related to the time period,
  • using Throttle.

If your requirement is to have a time period of 24 hours you have two solutions:

  • run your alert every 24 hours;
  • set a Throttle of 24 hours.

Anyway avoid Real Time.

Ciao.

Giuseppe

View solution in original post

0 Karma

marthin
Engager

Thanks Giuseppe. Yes well noted on the real-time, have changed it accordingly.

Regarding "Number of Results" (as applied in Rela-time searches) have tested a bit deeper into the behaviour with a high frequency log stream (one per second, is a protocol timetick) and found the following:

1. The key is the rolling window concept in the Real-time search. The window shifts and is not a discrete timeframe excluding previous matched events.

2. The way I had it setup it created an email every 5 seconds, with the last ~300 events in the mail.

Eg. Saved Search [SPLUNK ALERT - P3 - ESPMCASTPROBE B FEED]: number of events (299)

3. Received a mail once for each result that there were more than 10 events per search, which triggered every 5 seconds (12 per minute).

 

Having it now as a Scheduled search provides the result needed, ie.

1. Looking at the last 5 minutes

2. Evaluating if it has more than 10 events

3. Triggering the event accordingly

4. Scheduled search again for 5 mins ahead ("sleeping" for 5 mins)

 

many thanks

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @marthin,

good for you, tell me if I can still help you on this question, see next time!

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @marthin,

let me understand: you configured the alert as the screenshot and you have 33 emails instead 1, is it correct?

At first never use real time alerts because it consumes too much resources (every search takes one CPU and release it when finished, with real time it doesn't release CPU!).

Then configure Throttle: Throttle is a run exclusion when an alert is fired for a defined time, to avoid thet the alert is triggered many times with the same condition.

Then, using real time your condition is continously verified and the aler triggers.

The best approach is to schedule your alers e.g. every 5 minutes or every hour, frequency usually depends on the time period of a search: e.g. if I have a time period of 1 hours my alert runs every hour, it isn't correct to have an alert with a time period of 24 hours that runs continously because you have many fired alerts!

So I hint to re-design your alert:

  • reducing time period,
  • setting a frequency related to the time period,
  • using Throttle.

If your requirement is to have a time period of 24 hours you have two solutions:

  • run your alert every 24 hours;
  • set a Throttle of 24 hours.

Anyway avoid Real Time.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...