Getting Data In

"search head Timed out waiting for peer" How can I check the network health between a search head and indexer?

Path Finder

Our environment consists of 1 indexer and 1 search head. Our indexer is currently indexing close to 400GB per day, since we are catching up on historical data. In another week, this should reduce to about 20GB per day.
Meanwhile, we are running a few saved searches on the search head, which would normally run for a few hours. However, we always see the error:

Timed out waiting for peer xx-xxxx-xxx. If this occurs frequently, receiveTimeout in distsearch.conf may need to be increased. Search results might be incomplete!

I have increased receiveTimeout to 900s. I am planning on adding the following stanza to distsearch.conf to reduce the knowledge bundle size:

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

I know that there may be network issues that are causing the problem. Are there any commands I can use to check the network health between the search head and indexer?
Any other suggestions to avoid this message would be welcome.

1 Solution

Path Finder

It turns out the problem was the nature of the query itself. Since the query was searching for sparse events, there were cases where it would run for 900s, and actually have nothing to return.
Increasing the timeout to a much larger value solved the problem

View solution in original post

Path Finder

It turns out the problem was the nature of the query itself. Since the query was searching for sparse events, there were cases where it would run for 900s, and actually have nothing to return.
Increasing the timeout to a much larger value solved the problem

View solution in original post

Builder

Have you covered the basics with ping, traceroute, and telnet?

0 Karma