How to trouble shoot this ERROR TcpOutputFd - Connection to host=10.X.X.X failed ?

[Hemnaath]

Hi All, Currently we have been informed that two of the Windows domain server is not reporting as expected, so when checked the issue by executing the simple query in splunk we could see that the below error message.

index="_internal" source="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" log_level=ERROR host="test01"

11/27/17
10:54:31.463 AM
11-27-2017 10:54:31.463 -0500 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed
eventtype = err0r error eventtype = nix-all-logs eventtype = nix_errors error eventtype = splunkd-log host = test01 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd

connection to host=10.x.x.x is the HF instance

Kindly let me know how to fix this issue.

[mdsnmss]

Is your host=test01 one of the two Windows domain servers not reporting to the HF? Do you have other forwarders going to your HF that are working?

If nothing is currently going to the HF and everything is getting refused I would check that the HF is configured to receive and that port is open allowing data to flow through: http://docs.splunk.com/Documentation/Splunk/7.0.0/Forwarding/Enableareceiver. If some data is going through with other UFs to the HF I would ping the HF system from the system you are trying to forward from to make sure it can reach it. If that isn't an issue it could possibly be SSL configurations if you have SSL forwarding enabled.

I'd lean towards this either being an issue with inputs/outputs configurations or with ports. If the receiver is enabled and there are no firewall issues I would have to see your inputs/outputs to dig further.

[Hemnaath]

hi mdsnmss, hey thanks for responding on this issue, let me check the same and get back to you.

[Hemnaath]

hey issue got fixed after rebooting the servers, the server was in hung status so requested server team to reboot the server and it got fixed. But could you please suggest me some troubleshooting steps for resolving this type of Error message if it occurs in future.

thanks in advance.

[nickhills]

turn it off, and on again.
It worked last time 🙂

[mdsnmss]

If that doesn't work if it occurs again I'd still go down the road I suggested originally. I'll add in to check Splunk service first:

1.) Check Splunk service status
2.) Check that you can reach the HF from the UF attempting to send data. Try to ping the HF, check Firewall rules to make sure proper ports are open, and verify the HF is configure to listen.
3.) If enabled, verify SSL settings between UF and HF in inputs.conf and outputs.conf.