Getting Data In

Open TCP Connections from forwarders to indexer

dshakespeare_sp
Splunk Employee
Splunk Employee

When watching the network traffic from a Splunk Universal Forwarder to a Indexer there appears to be two connections.

There is a standard “data forwarding connection” (long lived, shows up with sourceIp and sourcePort in metrics.log under the tcpin_connections group) but there also appears to be a secondary connection opened by the forwarder. The second connection occurs every 30 seconds and lasts about 1 second after which it seemed to be closed by the forwarder.

What is the second connection used for? Does it indicate a problem with Splunk?

0 Karma

dshakespeare_sp
Splunk Employee
Splunk Employee

The secondary connection is a "healthcheck" probe.
The forwarder uses this connection to evaluate whether the receiver still in a "good working condition". When a receiver is effectively stalled so it is not processing more data, or
when it is in the process of shutting down, the receiver closes the TCP socket. A forwarder which cannot connect to the indexer's incoming socket will put the indexer into quarantine and not associate any more data streams with that indexer until the health check works again. This
allows Splunk to try to complete outgoing data streams, and ensures that if data need to be resent (eg useACK timeout) it would not pointlessly select this indexer even if we are already
connected to it.

dshakespeare_sp
Splunk Employee
Splunk Employee
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...