Splunk UF cant forward data to HF

[noobSpl888]

Hi,

ii had recently install UF v9.0.5 on our windows hosts to send logs to a heavy forwarder, and is getting below messages in the splunkd logs in windows host.

Can i know what are these info about?

ERROR TcpOutputFd [2404 TcpOutEloop] - Read error. An existing connection was forcibly closed by remote host

INFO AutoLoadBalancedConnectionStrategy [2404 TcpOutEloop] - Connection to 10.xx.xx.xx:9997 closed. Read error. An existing connection was forcibly closed by remote host

WARN AutoLoadBalancedConnectionStrategy [2404 TcpOutEloop] - Possibe duplication of events with channel=source::C:\Programs Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::xxxxx011|splunkd|2606, streamId=0, offset=0 on host=10.xx.xx.xx:9997

Thanks

[noobSpl888]

Hi,

 the outputs.conf is the same as in our linux hosts (which are working fine). anyway i shall check with my network team on the firewall level.  and also raise a support case.

thanks for the inputs. 

Cheers

[PickleRick]

OK, for the clarification. In my case (I have no idea if yours is the same but seems so). The RSTs were not sent as a response to the initial SYN - signalling a closed port. No, they were sent by the receiving end some time after the connection had been already established and some data was already sent through it. Just for no apparent reason the receiver decided that it wouldn't talk to the sender no more and would abruptly close the connection.

[PickleRick]

Ok, I've had a similar case but are you sure your events aren't getting sent to downstream? In my case they were and indeed duplication did occur.

Tl&dr - open a case with support.

You have two separate things here. One is a connection close. Unfortunately I didn't have time to dig too deply into it with the customer but it looks like a support ticket material.

As fat as I remember from looking at the network traffic, it was indeed the receiving side which suddenly was sending RSTs which was totally unexpected.

The other thing is that you probably have useAck enabled in your environment so as the UF tries to re-send the chunk of data it had in buffer when the connection was closed, it gets signaled that the downstream HF had already seen those because apparently closing the connection doesn't prevent the HF from processing the events further.

[gcusello]

Hi @noobSpl888,

there are three possible issues:

• the connection between UF and HF isn't open, maybe there's a firewall between them, check using telnet if it's open;
• you didn't enabled receiving on the HF, go in [Settings > Forwarding and Receiving > Receiving] and enable Receiving;
• you didn't point the correct address, how do you configured your outpts.conf?

Ciao.

Giuseppe