I must admit what is happening makes no sense. Take this error for example:
[OurIndexer01,OurIndexer02,OurIndexer03] Configuration initialization for C:\Program Files\Splunk\var\run\searchpeers\OurSearchHead01-1762950946 took longer than expected (10735ms) when dispatching a search with search ID remote_OurSearchHead01_1762951365.773. This might indicate an issue with underlying storage performance or the knowledge bundle size. If you want this message displayed more or less often, change the value of the 'search_startup_config_timeout_ms' setting in "limits.conf" to a lower or higher number.
that error says, this search took about 10 seconds and that's too long according to Splunk, well how can that be if in our limits.conf file on this Search Head we have the following stanza?
[search]
search_startup_config_timeout_ms = 30000
FYI we are on the latest version of Splunk 10.x
