Splunk searches skipped after upgrading to 8.1.0

[vagsec]

Hi all,

I have upgraded our Splunk index cluster from 7.3.0 to 8.1.0 and since then I see the below red message on search head:

The percentage of non high priority searches skipped (50%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=20. Total skipped Searches=10

Do you have any ideas how could I recover from this?  And what is causing it? I took all the steps as described here https://docs.splunk.com/Documentation/Splunk/8.1.0/Installation/AboutupgradingREADTHISFIRST 

I have followed this problem as well, but no luck: https://community.splunk.com/t5/Installation/Rolling-upgrade-restart-scheduled-searches-skipped-erro...

Regards,

Evang

Regards,

Evang

[SirDrake7]

@vagsec 

Did you ever get this resolved by chance?  I am having the same issue.

Thank you,

[vagnet]

Hi SirDrake7. I resolved it by increasing the maximum number of concurrent searches on the limits.conf file.

[SirDrake7]

Thank you,

Would you be able to share the section you added?

I ended up going in to Settings: Server Settings:  Search Preferences:  and I increased the Relative concurrency limit for scheduled searches and summarization searches from 50% to 100%.  Based off what supported stated my searches should not require any change over 50% - not to mention I had doubled my CPU's and upped my RAM x4 lol.  So if there is a config file change that would be better than my fix.