Deployment Architecture

UniversalForwarder 9.0.4 not able to connect to Enterprise splunk 8.0.4 indexer?

ball
Explorer

03-30-2023 01:56:34.810 -0400 INFO  AutoLoadBalancedConnectionStrategy [15424 TcpOutEloop] - Removing quarantine from idx=10.65.152.88:9997 connid=0

03-30-2023 01:56:34.811 -0400 WARN  TcpOutputFd [15424 TcpOutEloop] - Connect to 10.65.152.88:9997 failed. Connection refused

03-30-2023 01:56:34.811 -0400 ERROR TcpOutputFd [15424 TcpOutEloop] - Connection to host=10.65.152.88:9997 failed

03-30-2023 01:56:34.811 -0400 WARN  TcpOutputFd [15424 TcpOutEloop] - Connect to 10.65.152.88:9997 failed. Connection refused

 

what is the configuration issue at forwarded OR enterprise server level?

 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

It is a problem with connectivity. UF 9.x works perfectly OK with older indexers. It just generates configtracker inputs which are sent by default to a non-existent index. That results in events going into last-resort index or getting dropped and generating warnings. Other than that - there's no problem with UF 9 in Splunk 8 environment.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ball,

did you tested connection using telnet on the port 9997 from the UF to the Indexer?

telnet 10.65.152.88 9997

if route is closed check the firewall route.

If connection is open check if you're using SSL on port 9997.

Ciao.

Giuseppe

0 Karma

ball
Explorer

I don't have access to telnet so tested using ssh.

ssh -p 9997 user1@10.65.152.88

ssh: connect to host 10.65.152.88 port 9997: Connection refused.

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ball,

as @PickleRick said, SSH couldn't be the correct way to test connectivity, but the connection refused should demonstarte that there's a problem on Indexer open port, but if other UFs are sending logs it should be,

The most probable issue could be the firewall route.

Ciao.

Giuseppe

0 Karma

PickleRick
SplunkTrust
SplunkTrust

ssh is not a best tool for connectivity troubleshooting because it relies on a relatively high-level protocol for functioning. But still, "connection refused" shows network-level problems. Either closed ports on the destination machine or traffic not opened in between.

0 Karma

ball
Explorer

Other forwarder able to send to that indexer but not this ONLY.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Well, hard to say without knowing your environment. You might have host firewall rules prohibiting input. You might have network firewall blocking traffic. You might have limits on permitted IP addresses in inputs.conf on the indexer. There can be several things.

I'd simply run a tcpdump for the packets coming from the UF's IP and check if the packets appear on your network interface. This way you'll know if your problems seems to be on the network or on the host itself.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...