Open TCP Connections from forwarders to indexer

[dshakespeare_sp]

When watching the network traffic from a Splunk Universal Forwarder to a Indexer there appears to be two connections.

There is a standard “data forwarding connection” (long lived, shows up with sourceIp and sourcePort in metrics.log under the tcpin_connections group) but there also appears to be a secondary connection opened by the forwarder. The second connection occurs every 30 seconds and lasts about 1 second after which it seemed to be closed by the forwarder.

What is the second connection used for? Does it indicate a problem with Splunk?

[dshakespeare_sp]

The secondary connection is a "healthcheck" probe.
The forwarder uses this connection to evaluate whether the receiver still in a "good working condition". When a receiver is effectively stalled so it is not processing more data, or
when it is in the process of shutting down, the receiver closes the TCP socket. A forwarder which cannot connect to the indexer's incoming socket will put the indexer into quarantine and not associate any more data streams with that indexer until the health check works again. This
allows Splunk to try to complete outgoing data streams, and ensures that if data need to be resent (eg useACK timeout) it would not pointlessly select this indexer even if we are already
connected to it.

[dshakespeare_sp]

also see http://wiki.splunk.com/Community:Splunk_Socket_Behavior