Getting Data In

Open TCP Connections from forwarders to indexer

dshakespeare_sp
Splunk Employee
Splunk Employee

When watching the network traffic from a Splunk Universal Forwarder to a Indexer there appears to be two connections.

There is a standard “data forwarding connection” (long lived, shows up with sourceIp and sourcePort in metrics.log under the tcpin_connections group) but there also appears to be a secondary connection opened by the forwarder. The second connection occurs every 30 seconds and lasts about 1 second after which it seemed to be closed by the forwarder.

What is the second connection used for? Does it indicate a problem with Splunk?

0 Karma

dshakespeare_sp
Splunk Employee
Splunk Employee

The secondary connection is a "healthcheck" probe.
The forwarder uses this connection to evaluate whether the receiver still in a "good working condition". When a receiver is effectively stalled so it is not processing more data, or
when it is in the process of shutting down, the receiver closes the TCP socket. A forwarder which cannot connect to the indexer's incoming socket will put the indexer into quarantine and not associate any more data streams with that indexer until the health check works again. This
allows Splunk to try to complete outgoing data streams, and ensures that if data need to be resent (eg useACK timeout) it would not pointlessly select this indexer even if we are already
connected to it.

dshakespeare_sp
Splunk Employee
Splunk Employee
0 Karma
Get Updates on the Splunk Community!

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...

September Community Champions: A Shoutout to Our Contributors!

As we close the books on another fantastic month, we want to take a moment to celebrate the people who are the ...

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...