Configuration initialization took longer than expected when dispatching a search

[Gregski11]

I must admit what is happening makes no sense. Take this error for example:

[OurIndexer01,OurIndexer02,OurIndexer03] Configuration initialization for C:\Program Files\Splunk\var\run\searchpeers\OurSearchHead01-1762950946 took longer than expected (10735ms) when dispatching a search with search ID remote_OurSearchHead01_1762951365.773. This might indicate an issue with underlying storage performance or the knowledge bundle size. If you want this message displayed more or less often, change the value of the 'search_startup_config_timeout_ms' setting in "limits.conf" to a lower or higher number.

that error says, this search took about 10 seconds and that's too long according to Splunk, well how can that be if in our limits.conf file on this Search Head we have the following stanza?

[search]
search_startup_config_timeout_ms = 30000

FYI we are on the latest version of Splunk 10.x

[nyc_jason]

Hello Gregski11. The timeout setting simple tells splunk to warn you about it, not to disallow the search. the default is 3000ms, so check that your local limits is even readable by splunk. There could be several reasons for the search taking so long to dispatch, such as permission on the directory where the artifacts are created, or even on you limits.conf in your local (if splunk cant read it, it will use the default version).  if the disk where the artifacts are create is very slow for some reason, then it could just be taking a while to create. 

[PickleRick]

Another notorious cause for huge knowledge bundle is when someone does a big search and happily ends it with | outputlookup. I've seen several gigabytes of data dragged pointlessly back and forth because of that.