topic Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer in Deployment Architecture

UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer?

ball — Thu, 30 Mar 2023 15:08:51 GMT

03-30-2023 01:56:34.810 -0400 INFO AutoLoadBalancedConnectionStrategy [15424 TcpOutEloop] - Removing quarantine from idx=10.65.152.88:9997 connid=0

03-30-2023 01:56:34.811 -0400 WARN TcpOutputFd [15424 TcpOutEloop] - Connect to 10.65.152.88:9997 failed. Connection refused

03-30-2023 01:56:34.811 -0400 ERROR TcpOutputFd [15424 TcpOutEloop] - Connection to host=10.65.152.88:9997 failed

03-30-2023 01:56:34.811 -0400 WARN TcpOutputFd [15424 TcpOutEloop] - Connect to 10.65.152.88:9997 failed. Connection refused

what is the configuration issue at forwarded OR enterprise server level?

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

gcusello — Thu, 30 Mar 2023 10:58:17 GMT

Hi @ball,

did you tested connection using telnet on the port 9997 from the UF to the Indexer?

telnet 10.65.152.88 9997

if route is closed check the firewall route.

If connection is open check if you're using SSL on port 9997.

Ciao.

Giuseppe

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

PickleRick — Thu, 30 Mar 2023 11:02:58 GMT

It is a problem with connectivity. UF 9.x works perfectly OK with older indexers. It just generates configtracker inputs which are sent by default to a non-existent index. That results in events going into last-resort index or getting dropped and generating warnings. Other than that - there's no problem with UF 9 in Splunk 8 environment.

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

ball — Thu, 30 Mar 2023 12:14:25 GMT

I don't have access to telnet so tested using ssh.

ssh -p 9997 user1@10.65.152.88

ssh: connect to host 10.65.152.88 port 9997: Connection refused.

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

PickleRick — Thu, 30 Mar 2023 12:31:00 GMT

ssh is not a best tool for connectivity troubleshooting because it relies on a relatively high-level protocol for functioning. But still, "connection refused" shows network-level problems. Either closed ports on the destination machine or traffic not opened in between.

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

gcusello — Thu, 30 Mar 2023 12:36:52 GMT

Hi @ball,

as @PickleRick said, SSH couldn't be the correct way to test connectivity, but the connection refused should demonstarte that there's a problem on Indexer open port, but if other UFs are sending logs it should be,

The most probable issue could be the firewall route.

Ciao.

Giuseppe

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

ball — Thu, 30 Mar 2023 16:22:17 GMT

Other forwarder able to send to that indexer but not this ONLY.

Re: UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer

PickleRick — Thu, 30 Mar 2023 19:15:32 GMT

Well, hard to say without knowing your environment. You might have host firewall rules prohibiting input. You might have network firewall blocking traffic. You might have limits on permitted IP addresses in inputs.conf on the indexer. There can be several things.

I'd simply run a tcpdump for the packets coming from the UF's IP and check if the packets appear on your network interface. This way you'll know if your problems seems to be on the network or on the host itself.