Getting Data In

"search head Timed out waiting for peer" How can I check the network health between a search head and indexer?

nivedita_viswan
Path Finder

Our environment consists of 1 indexer and 1 search head. Our indexer is currently indexing close to 400GB per day, since we are catching up on historical data. In another week, this should reduce to about 20GB per day.
Meanwhile, we are running a few saved searches on the search head, which would normally run for a few hours. However, we always see the error:

Timed out waiting for peer xx-xxxx-xxx. If this occurs frequently, receiveTimeout in distsearch.conf may need to be increased. Search results might be incomplete!

I have increased receiveTimeout to 900s. I am planning on adding the following stanza to distsearch.conf to reduce the knowledge bundle size:

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

I know that there may be network issues that are causing the problem. Are there any commands I can use to check the network health between the search head and indexer?
Any other suggestions to avoid this message would be welcome.

1 Solution

nivedita_viswan
Path Finder

It turns out the problem was the nature of the query itself. Since the query was searching for sparse events, there were cases where it would run for 900s, and actually have nothing to return.
Increasing the timeout to a much larger value solved the problem

View solution in original post

nivedita_viswan
Path Finder

It turns out the problem was the nature of the query itself. Since the query was searching for sparse events, there were cases where it would run for 900s, and actually have nothing to return.
Increasing the timeout to a much larger value solved the problem

brodriguez
Splunk Employee
Splunk Employee

Could you please specify which timeout setting did you increase?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You're digging up an 8 years old thread. I wouldn't expect an answer from its original participants...

0 Karma

dflodstrom
Builder

Have you covered the basics with ping, traceroute, and telnet?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...