Deployment Architecture

Broken Pipe

skuma30
New Member

Hi,

Recently we are experiencing broken pipe errors we never had this issue before but all of a sudden it is showing in the splunkd.logs and indexer peer is not connecting to the search head. Due to this the search head is going to idle till the dispatch time reaches and very slow search performance.

error log:-
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe
01-09-2018 16:39:37.374 -0600 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:xx. Broken pipe

Tags (1)
0 Karma

mayurr98
Super Champion

This error is caused by the heartbeat function. every 30 seconds the heartbeat will send to indexer. if the indexer don't get it during that time, the indexer writes a log with the timeout message. network devices like a firewall can causing this or long remote connections. I disabled the heartbeat. Other solution could be change the time frequency from 30 seconds...

heartbeatFrequency =
How often (in seconds) to send a heartbeat packet to the receiving server.
Heartbeats are only sent if sendCookedData=true.
* Defaults to 30 (seconds).

It is a mechanism for the forwarder to to know that the receiver (ie indexer) is alive. If the indexer does not send a return packet to the forwarder, the forwarder will declare this receiver unreachable and not forward data to it. By default a packet is sent every 30s.

Thank you

Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...