What is the maximum amount of alerts can you have running in real time?

[splunktrainingu]

Hello,

I have been making a ton of alerts to notify me on certain event IDs but this morning when I logged into my console I saw a huge red exclamation mark next to "Searches Skipped." I looked at the alert and it said "red threshold has been exceed" "40% of searches were skipped". I disabled 7 of the alerts and consolidated two alerts into one.  The issue went away. My question is how can I monitor for those event IDs in real time without triggering  the threshold. 

[richgalloway]

Don't use real-time alerts.  Humans can't respond to them fast enough to make them worthwhile.

The number of searches that can run at any given time is governed by the number of CPUs in the search head.  To get more capacity, add CPUs or expand your SHC.  Note that your indexer layer needs sufficient CPUs to process the searches, too, as well as index incoming data.

See if you can consolidate more alerts.

[splunktrainingu]

Well what do you recommend I should use if not real time alerts? I guess I could put together the events that are not likely to happen and use  a scheduled search to run maybe run every 6 hours.  But what I was trying to achieve with splunk enterprise is real time alerting.

[richgalloway · Accepted solution]

Real-time alerting is only useful when you have real-time responses to those alerts.  Otherwise, schedule them for every 5 or 10 minutes.  That's still often enough to catch problems early without tying up a CPU like a real-time search does.