Splunk Universal Forwarder- Why are some of my hosts not sending to indexer?

[Darsh1561]

My network has 34 hosts with universal forwarder setup on each of them but only 5 of them are forwarding their logs to indexer.

So what shall I do to overcome this problem ?

PS. The firewall on the indexer has already been turned off, to avoid any package drop.

[PickleRick]

As your network-level connectivity seems to be OK (at least that's what you said), you have to check what the forwarder reports.

Check your $SPLUNK_HOME/var/log/splunk/splunkd.log and see what entries you have at/near the end of the file.

If you have entries similar to this:

01-25-2023 08:45:31.227 +0100 INFO AutoLoadBalancedConnectionStrategy [1500 TcpOutEloop] - Connected to idx=172.16.0.3:9997:0, pset=0, reuse=0. autoBatch=1

or this:

01-29-2023 12:18:13.357 +0100 INFO AutoLoadBalancedConnectionStrategy [1500 TcpOutEloop] - Found currently active indexer. Connected to idx=172.16.0.3:9997:0, reuse=1.

(of course with your indexer's IP), it means that the forwarder connects to donwstream indexers but for some reason the data is indistinguishable from other forwarders.

If you have some connection problems instead, that means that for some reason your forwarder can't connect and you should have the reason why in the log.

[gcusello]

Hi @Darsh1561,

at first check the connection with the Indexer using telnet on port 9997.

Then share one information: did you installed each Universal Forwarder one by one or cloning an installation?

If cloning, check the hostname in $SPLUNK_HOME/etc/syetm/local7server.conf

Ciao.

Giuseppe

[Darsh1561]

I checked for the connection at port 9997 using telnet which is active.

I have individually installed forwarder on each host. (i.e did not clone it)

[gcusello]

Hi @Darsh1561,

when you say that only 5 of 34 UFs are forwarding logs, are you speaking of server logs or also of Splunk internal logs? you can check this on _internal index.

How do you manage these forwarders? manually or are you using a Deployment Server?

outputs.conf is the same in all UFs?

plese try to restart Splunk on one of the missing UFs.

Ciao.

Giuseppe

[Darsh1561]

I am speaking of both server and internal logs. I am managing UF manually only.

The outputs.conf is same in all UFs.

I already tried restarting and re-installing but nothing is working.

[gcusello]

Hi @Darsh1561,

please check if the hostname in $SPLUNK_HOME/etc/system/local/server.conf is different in each UF and is the same of the OS.

then check in $SPLUNK_HOME/var/log/splunk/splunkd.log of the UF eventual connection refused messages.

Ciao.

Giuseppe

[Darsh1561]

I checked the log file it says failed to complete handshake.

[gcusello]

Hi @Darsh1561,

are you using SSL to protect your connection?

Ciao.

Giuseppe

[Darsh1561]

No, I ain't using ssl