I recently noticed that two separate splunk instances were reporting that around 99% of their scheduled searches were being skipped, and after further research, I found that almost all of the skipped searches had to do with cisco security suite.
What could be causing this type of behavior? I found another thread where someone asked the same question, but it does not look like they ever came to a solid conclusion.
I thought this might be due to hardware resources at first, but the one machine has 32 CPU cores and 12 gigs of ram that aren't anywhere close to being fully utilized, so I don't think it has anything to do with that.
I've got this problem, too. Disabling searches feels like a workaround. It's kinda hard to tell which ones you "need" and which you don't. You don't, for instance, need a saved search about a WSA if you don't have a WSA. But your "Top Attackers" search doesn't matter until you get attacked.
I ended up disabling and un-accelerating un-needed saved searches (76 of them). In a local instance of savedsearches.conf:
/opt/splunk/etc/deployment-apps/Splunk_CiscoSecuritySuite/local/savedsearches.conf
[Cisco WSA - Web Request Metrics - Users with Multiple UAs]
disabled = 1
auto_summarize = 0
.
.
.
[Cisco IPS - GC - Top Attackers]
disabled = 1
auto_summarize = 0
Commenting to say I'm having the same issue. No warnings generated like "real" concurrency maximums. Every other search has a 0% skip ratio, but Cisco Security Suite is at 98-99% stating ridiculous and impossible things like:
The maximum number of concurrent auto-summarization searches on this instance has been reached (1008) The maximum number of concurrent historical scheduled searches on this instance has been reached (109)
My concurrent max searches is something around 13, so I have no idea where it's getting the 1000+ and 100+ numbers. The monitoring console (outside of the scheduler activity) shows no issue with any searching, pipeline, etc.
Anything new on this? Got the same problem.
If it is a real-time search then it will continuously show as "skipped" because it never stops and the scheduler will not start the same search if the previous window's search is still running. Is it real-time
?
Hi,
Thanks for your help, would real-time be the same thing as an accelerated search?
It shouldn't be. Did you ever figure this out?