Alerting

What is the maximum amount of alerts can you have running in real time?

splunktrainingu
Communicator

Hello,

I have been making a ton of alerts to notify me on certain event IDs but this morning when I logged into my console I saw a huge red exclamation mark next to "Searches Skipped." I looked at the alert and it said "red threshold has been exceed" "40% of searches were skipped". I disabled 7 of the alerts and consolidated two alerts into one.  The issue went away. My question is how can I monitor for those event IDs in real time without triggering  the threshold. 

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Real-time alerting is only useful when you have real-time responses to those alerts.  Otherwise, schedule them for every 5 or 10 minutes.  That's still often enough to catch problems early without tying up a CPU like a real-time search does.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Don't use real-time alerts.  Humans can't respond to them fast enough to make them worthwhile.

The number of searches that can run at any given time is governed by the number of CPUs in the search head.  To get more capacity, add CPUs or expand your SHC.  Note that your indexer layer needs sufficient CPUs to process the searches, too, as well as index incoming data.

See if you can consolidate more alerts.

---
If this reply helps you, Karma would be appreciated.

splunktrainingu
Communicator

Well what do you recommend I should use if not real time alerts? I guess I could put together the events that are not likely to happen and use  a scheduled search to run maybe run every 6 hours.  But what I was trying to achieve with splunk enterprise is real time alerting.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Real-time alerting is only useful when you have real-time responses to those alerts.  Otherwise, schedule them for every 5 or 10 minutes.  That's still often enough to catch problems early without tying up a CPU like a real-time search does.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...