topic Re: Connectivity between depolyment client and indexer in Deployment Architecture

Connectivity between depolyment client and indexer

mship — Mon, 14 Jan 2013 13:54:05 GMT

I am receiving the following message in the splunkd log on my UF (windows)

01-11-2013 09:47:35.129 -0500 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed
01-11-2013 09:47:35.129 -0500 WARN TcpOutputProc - Applying quarantine to idx=x.x.x.x:9997 numberOfFailures=2
01-11-2013 09:48:04.141 -0500 INFO TcpOutputProc - Removing quarantine from idx=x.x.x.x:9997
01-11-2013 09:48:05.072 -0500 WARN TcpOutputFd - Connect to x.x.x.x:9997 failed. No connection could be made because the target machine actively refused it.

I can ping between the indexer and UF and nothing seems amiss. I cannot test with telnet b/c it is disabled. Any suggestions for troubleshooting is appreciated! Thanks!

Re: Connectivity between depolyment client and indexer

Ayn — Mon, 14 Jan 2013 13:59:49 GMT

In order to properly troubleshoot the issue you need some kind of tool to check the connection to port 9997 on the indexer. This kind of problem is very often linked to firewall issues, or configuration issues on the indexer (port 9997 is not configured to receive data from other Splunk instances). Do you have other instances that are working properly and it's just an issue with this specific instance?

Re: Connectivity between depolyment client and indexer

mship — Mon, 14 Jan 2013 14:25:04 GMT

Thanks Ayn. I will see if I cant get something to test it out. Peculiar thing is that everything was working fing until 2 days ago and to my knowledge nothing has changed.

Re: Connectivity between depolyment client and indexer

mship — Wed, 16 Jan 2013 15:35:23 GMT

Ayn the issue is with all instances. I have re-enabled telnet on the network and I can telnet to and from client/server over port 23 but I cannot over port 9997. It seems that when I restart the splundd a few times I get a full update and then it all stops again so I know it can connect at some point but then it is refused...

Re: Connectivity between depolyment client and indexer

mship — Wed, 16 Jan 2013 17:02:03 GMT

Update: I can now telnet to and from client/server over 9997 but still not receiving data.

Re: Connectivity between depolyment client and indexer

Ayn — Wed, 16 Jan 2013 21:11:18 GMT

...and are you seeing the same error messages in your UF logs still?

Re: Connectivity between depolyment client and indexer

mship — Fri, 18 Jan 2013 15:30:21 GMT

Yup...this morning I added the following entry to the inputs.conf file on the indexer

[splunktcp://9997] connection_host = none

I now seem to be receiving data...does this make sense to you?

Re: Connectivity between depolyment client and indexer

mship — Wed, 23 Jan 2013 16:38:35 GMT

I solved the issue...

I added the following line to the /etc/system/local/inputs.conf file on the indexer
[splunktcp://9997]
Connection_host = none

I ran this by splunk support and they indicated that this is a good fix and that they are experiencing a bug in DNS reverse lookups and this is a good workaround.

Re: Connectivity between depolyment client and indexer

lrodriguez_splu — Mon, 25 Mar 2019 23:54:05 GMT

This worked for me!