I believe the bottom line is that the resources need to be there on the indexer so that there are minimal to no skips on searches. The ideal Splunk configuration is that you have an indexer and a search head. The search head does what it does and if there are enough resources there would be no skipping searches. This way the load is kept separate. So if you have an indexer/search head on one server, there needs to be a lot of resources as it is searches per physical core, not virtual and the more you have with more memory then the probability of skipped searches go down. If that is not the case, and there is no way around the physical cpu and memory issues, then what I have found is that you can modify the limits.conf file. It is not something that Splunk support recommends, but it helps. Here are the places that I have changed:
[scheduler]
max_searches_perc = 50 (default)
max_searches_per_cpu = 1 (default)
I changed the max_searches_perc to 60 and max_searches_per_cpu to 10 to see if the skipping searches would go back to 0.00%. When it did I slowly lowered it down until I found a good point where there may be a small percentage skipped or none at all. I also changed the max_searches_perc back down to 50 at that point and watched it. With all the apps that I have and what I need it to do,Right now the max searches per cpu is at 5. I am good with that. I may have to go to getting another server for a search head.
I hope this helps.
... View more