Installation

Splunk Universal Forwarder- Why are some of my hosts not sending to indexer?

Darsh1561
Explorer

My network has 34 hosts with universal forwarder setup on each of them but only 5 of them are forwarding their logs to indexer.

So what shall I do to overcome this problem ?

PS. The firewall on the indexer has already been turned off, to avoid any package drop.

Labels (4)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

As your network-level connectivity seems to be OK (at least that's what you said), you have to check what the forwarder reports.

Check your $SPLUNK_HOME/var/log/splunk/splunkd.log and see what entries you have at/near the end of the file.

If you have entries similar to this:

01-25-2023 08:45:31.227 +0100 INFO AutoLoadBalancedConnectionStrategy [1500 TcpOutEloop] - Connected to idx=172.16.0.3:9997:0, pset=0, reuse=0. autoBatch=1

or this:

01-29-2023 12:18:13.357 +0100 INFO AutoLoadBalancedConnectionStrategy [1500 TcpOutEloop] - Found currently active indexer. Connected to idx=172.16.0.3:9997:0, reuse=1.

(of course with your indexer's IP), it means that the forwarder connects to donwstream indexers but for some reason the data is indistinguishable from other forwarders.

If you have some connection problems instead, that means that for some reason your forwarder can't connect and you should have the reason why in the log.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Darsh1561,

at first check the connection with the Indexer using telnet on port 9997.

Then share one information: did you installed each Universal Forwarder one by one or cloning an installation?

If cloning, check the hostname in $SPLUNK_HOME/etc/syetm/local7server.conf

Ciao.

Giuseppe

0 Karma

Darsh1561
Explorer

I checked for the connection at port 9997 using telnet which is active.

I have individually installed forwarder on each host. (i.e did not clone it)

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Darsh1561,

when you say that only 5 of 34 UFs are forwarding logs, are you speaking of server logs or also of Splunk internal logs? you can check this on _internal index.

How do you manage these forwarders? manually or are you using a Deployment Server?

outputs.conf is the same in all UFs?

plese try to restart Splunk on one of the missing UFs.

Ciao.

Giuseppe

0 Karma

Darsh1561
Explorer

I am speaking of both server and internal logs. I am managing UF manually only.

The outputs.conf is same in all UFs.

I already tried restarting and re-installing but nothing is working.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Darsh1561,

please check if the hostname in $SPLUNK_HOME/etc/system/local/server.conf is different in each UF and is the same of the OS.

then check in $SPLUNK_HOME/var/log/splunk/splunkd.log of the UF eventual connection refused messages.

Ciao.

Giuseppe

0 Karma

Darsh1561
Explorer

I checked the log file it says failed to complete handshake.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Darsh1561,

are you using SSL to protect your connection?

Ciao.

Giuseppe

0 Karma

Darsh1561
Explorer

No, I ain't using ssl

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...