Getting Data In

Event latency due to Intermittent forwarder & indexer timed out issues

Splunk Employee
Splunk Employee

Our site has 20+ indexers and 5000+ forwarders. The indexers and forwarders are on splunk 6.0.x.

We are suspecting that TcpInputProc on indexer might have been stalled in mid-flight, and would eventually recovered; forwarding would resume again. However, due this issue, we have been experiencing latency of events being sent to indexer. We have also confirmed that all the indexers were in "healthy" state. No signs of index congestion or high memory or cpu utilization.

splunkd.log of indexers report:

ERROR TcpInputProc - Error encountered for connection from src= Timeout Timed out after 600 seconds.

splunkd.log of forwarders report any number of following messaages:

ERROR TcpOutputFd - Connection to host= failed
WARN TcpOutputFd - Connect to failed. Connection refused
WARN TcpOutputProc - Cooked connection to ip= timed out

But this
WARN  TcpOutputProc - Cooked connection to ip= timed out
seems to be more frequently reported than the other messages.

Splunk Employee
Splunk Employee

The symptoms described above pertains to a known bug: SPL-84550: [splunktcp://] input does not inherit default setting "connection_host = ip" from [splunktcpin] stanza, leads to intermittent forwarder connection timeouts.

The root cause seems to be that splunk indexer (specifically the splunktcpin acceptor thread) is stuck in performing reverse DNS resolution for a handful of forwarders, and refusing new forwarder connection requests, or allowing some forwarder connections to time out (default forwarder connection time out is 20 sec).

Workaround solution is to set connection_host = ip in indexers' inputs.conf. Example:

connection_host = ip

Splunk Employee
Splunk Employee

SPL-84550 is expected to be fixed in an upcoming maintenance release beyond 6.0.5

SPL-85899 is the 6.1.x version of the bug and fixed in the currently available 6.1.3 maintenance release.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...