I have upgraded our Splunk index cluster from 7.3.0 to 8.1.0 and since then I see the below red message on search head:
The percentage of non high priority searches skipped (50%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=20. Total skipped Searches=10
Do you have any ideas how could I recover from this? And what is causing it? I took all the steps as described here https://docs.splunk.com/Documentation/Splunk/8.1.0/Installation/AboutupgradingREADTHISFIRST
I have followed this problem as well, but no luck: https://community.splunk.com/t5/Installation/Rolling-upgrade-restart-scheduled-searches-skipped-erro...
Would you be able to share the section you added?
I ended up going in to Settings: Server Settings: Search Preferences: and I increased the Relative concurrency limit for scheduled searches and summarization searches from 50% to 100%. Based off what supported stated my searches should not require any change over 50% - not to mention I had doubled my CPU's and upped my RAM x4 lol. So if there is a config file change that would be better than my fix.