How do I determine the appropriate timeout value for distributed search

[matt]

I see timeouts with distributed search and currently have receiveTimeout set to 300. What metric(s) can I use to determine the what should be the correct value this setting in order to avoid timeouts?

[rstrong30]

Some searches are going back to "ALL TIME" so I'd imagine there is an appropriate time to change the default. Depending on one's environment that could go back quite a few months. So if the default is 600 seconds I would imagine changing this to a few more minutes should be safe, right? If not, what's the repercussions?

[hexx · Accepted solution]

The current default value for this parameter is 600 seconds and there really shouldn't be any good reason to proactively increase or lower it.

If while searching, you see errors reporting search peer timeouts, the right question to ask is "Why are search peers taking more than 600 seconds to respond?".

• The first place to look for a possible explanation is the search.log file created by the search process on the search-head, in $SPLUNK_HOME/var/run/splunk/dispatch/{SID} where {SID} is the ID of the search.

• Another good place to look is the search.log file created by the search process on the remote peer, in $SPLUNK_HOME/var/run/splunk/dispatch/remote_{SH-ServerName}_{SID} where {SH-ServerName} is the ServerName as set in server.conf for the search-head that dispatched the search, and {SID} is the ID of the search.

  • As peer timeouts are often linked with network quality issues, it is also a good idea to check that there are no issues with the network link between the search-head and the affected peer(s).

[jdunlea_splunk]

I also need to find this out