Broken Pipe

[skuma30]

Hi,

Recently we are experiencing broken pipe errors we never had this issue before but all of a sudden it is showing in the splunkd.logs and indexer peer is not connecting to the search head. Due to this the search head is going to idle till the dispatch time reaches and very slow search performance.

error log:-
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe

[mayurr98]

This error is caused by the heartbeat function. every 30 seconds the heartbeat will send to indexer. if the indexer don't get it during that time, the indexer writes a log with the timeout message. network devices like a firewall can causing this or long remote connections. I disabled the heartbeat. Other solution could be change the time frequency from 30 seconds...

heartbeatFrequency =
How often (in seconds) to send a heartbeat packet to the receiving server.
Heartbeats are only sent if sendCookedData=true.
* Defaults to 30 (seconds).

It is a mechanism for the forwarder to to know that the receiver (ie indexer) is alive. If the indexer does not send a return packet to the forwarder, the forwarder will declare this receiver unreachable and not forward data to it. By default a packet is sent every 30s.

Thank you