Connectivity between depolyment client and indexer

[mship]

I am receiving the following message in the splunkd log on my UF (windows)

01-11-2013 09:47:35.129 -0500 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed
01-11-2013 09:47:35.129 -0500 WARN TcpOutputProc - Applying quarantine to idx=x.x.x.x:9997 numberOfFailures=2
01-11-2013 09:48:04.141 -0500 INFO TcpOutputProc - Removing quarantine from idx=x.x.x.x:9997
01-11-2013 09:48:05.072 -0500 WARN TcpOutputFd - Connect to x.x.x.x:9997 failed. No connection could be made because the target machine actively refused it.

I can ping between the indexer and UF and nothing seems amiss. I cannot test with telnet b/c it is disabled. Any suggestions for troubleshooting is appreciated! Thanks!

[mship]

I solved the issue...

I added the following line to the /etc/system/local/inputs.conf file on the indexer
[splunktcp://9997]
Connection_host = none

I ran this by splunk support and they indicated that this is a good fix and that they are experiencing a bug in DNS reverse lookups and this is a good workaround.

[lrodriguez_splu]

This worked for me!

[mship]

Yup...this morning I added the following entry to the inputs.conf file on the indexer

[splunktcp://9997] connection_host = none

I now seem to be receiving data...does this make sense to you?

[Ayn]

...and are you seeing the same error messages in your UF logs still?

[mship]

Update: I can now telnet to and from client/server over 9997 but still not receiving data.

[mship]

Ayn the issue is with all instances. I have re-enabled telnet on the network and I can telnet to and from client/server over port 23 but I cannot over port 9997. It seems that when I restart the splundd a few times I get a full update and then it all stops again so I know it can connect at some point but then it is refused...

[mship]

Thanks Ayn. I will see if I cant get something to test it out. Peculiar thing is that everything was working fing until 2 days ago and to my knowledge nothing has changed.

[Ayn]

In order to properly troubleshoot the issue you need some kind of tool to check the connection to port 9997 on the indexer. This kind of problem is very often linked to firewall issues, or configuration issues on the indexer (port 9997 is not configured to receive data from other Splunk instances). Do you have other instances that are working properly and it's just an issue with this specific instance?