I found below on Reditt which fixed my issue: https://www.reddit.com/r/Splunk/comments/kx2bo6/splunk_searches_skipped_error/gjaak2e?utm_source=share&utm_medium=web2x&context=3 I'm a little late to responding here but I just went thru this scenario myself. I found anytime we made any change it would cause searches to be delayed until they caught up. Before resorting to upgrading the limits.conf you need to identify what change caused Splunk to get overwhelmed. Since things are not working for you this may be a little difficult. I would recommend contacting support if you’re not comfortable with this. Check Correlation searches and ensure they are not set to real-time search. This would consume resources nonstop. Check Data Models and disable acceleration on any you may not be using. Verify time frame on the data models you leave accelerated. I found some apps had acceleration enabled with a long backfile range and that would take up a large amount of resources until it catches up. I did end up working with support and received some real good clarification on CPU settings for limits.conf. Let me first clarify the confusion: 1 CPU can have 16 cores. Splunk suggest 1 search per CPU core not per CPU. The defaults are set extremely low. I’m sure that’s something that supposed to get updated during onboarding of Splunk with some professional services. You will have to look at what CPU’s you have and find out how many cores each one has. base_max_searches = default 6*(let max search per cpu do the work) max_searches_per_cpu = if 1 CPU has 16 cores make sure to leave some room for overhead processes. So 12 would be a sweet spot. max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches (example: 12 x 16 + 6 = 198) do not modify this After making these changes I have not had this occur again. I've monitored CPU utilization and it has remained stable. Again, only do the limits.conf changes once you have figure out what is taking up so many resources. As that may just cause the server to constantly use a lot of resources. This must be configured on the search head and indexers. Once you get this working I do recommend using the SplunkAdmins to help identify further issues. There could be a large amount of underlying issues you may not even be aware of.
... View more
