I'm seeing an error related to HTTPStreamPerf - read failed. What does this mean?

[the_wolverine]

I'm seeing the following errors in Splunk in my distributed search environment. What do they mean and how can I fix this?

WARN  NetUtils - Bad select_for_loop rv = -2
ERROR HTTPStreamPerf - read failed. error=-2
DispatchCommand - Read 0 results from result provider peername=hostname.domain.com at uri=https://ip_addr:8089 timetaken=30.00

[the_wolverine · Accepted solution]

Most likely you're hitting a timeout on the searchhead. The "timetaken=30.00" is a clue. The default receiveTimeout value is 30 (seconds) so increasing this value may resolve the situation. The value can be changed in $SPLUNK_HOME/etc/system/local/distsearch.conf:

[distributedSearch]
receiveTimeout = 120

In Splunk version 4.1.4, we will display a banner error in UI indicating that the receiveTimeout was reached.