Getting Data In

Universal Forwarder

butlerm494
New Member

Before I start this is a serious case of blind leading the blind.

Currently we have a VMware running Windows Server 2016 hosting Splunk Enterprise, to date we have managed to get the forwarder installed on Windows 7, Windows 2003, Windows 2008, Solaris and Mint Linux (Just for a laugh). Without much administration it all works well, but we come to RHEL 7, for some reason we can not get it work, everything appears to be okay. Installed the RPM forwarder, but nothing appears to be happening.

As this is a test system we have disabled both Server and Client firewalls, can ping the server in both directions - but we can't seem to get it to work. The only thing that we have managed to find using "google" is a potential issue with SELINUX so we have disabled that.

Any suggestions as this would save the sanity of the "intern"

0 Karma

xpac
SplunkTrust
SplunkTrust

Try this on the command line:

/opt/splunkforwarder/bin/splunk list forward-server

It should show you if the UF has successfully connected to any configured destination server.

Also, do you get ANY logs from the forwarder at all, if only _internal logs?

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

0 Karma

butlerm494
New Member

Thank you for your comment,
I tried that command and it lists the ip address of the machine on which splunk enterprise is installed under active forwards.
Any other suggestion is appriciated

0 Karma

xpac
SplunkTrust
SplunkTrust

On your search head/indexer, see if you get ANY data of that forwarder (you should, if the indexer shows up as "active forwards"). Try it like this:

| tstats prestats=t count where (index=* OR index=_*) AND host=yourforwardername by _time index
| timechart count by index
0 Karma

p_gurav
Champion

Is there any error in _internal logs in /opt/splunkforwarder/var/log/splunk/splunkd.log?

butlerm494
New Member

I have TcpOutputFd - read error. Connection reset by peer
tcpoutputProc - connection to closed. Read error. Connection reset by peer.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Can you post your inputs.conf stanza? Also, what do the forwarder log files say?

You can go to /opt/splunkforwarder/var/log/splunk/splunkd.log and take a look

0 Karma

butlerm494
New Member

Thank you for your answer.
All I have in the inputs.conf file is
[default]
host = localhost.localdomain.
I can't post the log file since the system is on a standalone machine.
Any suggestion is welcome

0 Karma

xpac
SplunkTrust
SplunkTrust

If you don't have anything else in your inputs.conf, you simply didn't setup any inputs. There is no data going to come because you didn't tell it what to collect. 😉

0 Karma

butlerm494
New Member

but splunk enterprise should still be able to see the forwarder right? Instead I have no clients phoning home.

0 Karma

xpac
SplunkTrust
SplunkTrust

No, not by default. You need to configure your UF with the IP of the deployment server, they don't call home by default.
You could do this by doing /opt/splunkforwarder/bin/splunk set deploy-poll YOURSERVER:8089 (and maybe restarting).
You could also create a seperate app (this is the clean way!) with a deploymentclient.conf like this:

```
[deployment-client]

[target-broker:deploymentServer]
targetUri= YOURSERVER:8089
```

YOURSERVER has to be replaced with the IP or DNS name of your Splunk instance.

0 Karma

butlerm494
New Member

I have also already set the deploy-poll and restarted afterwards. The forwarder still does not appear on splunk enterprise

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Once again, you need to see what the forwarder logs are saying to troubleshoot your issue.. You claimed to install the UF on a RHEL server, so you can either look on that RHEL server under the path I gave you above, or if you are forwarder your UF log files, you can look in Splunk. We are unable to help you until you look

0 Karma

xpac
SplunkTrust
SplunkTrust

Can you use tcpdump on the Splunk Enterprise instance to check if you get any communication from that instance to TCP port 9997?
Also, did you try to check for any logs with my tstats command posted in the other comment?

0 Karma

butlerm494
New Member

I can't post the logs since they are on a standalone system, all we have on the universal forwarder inputs.conf is
[default]
host = localhost.localdomain
Any suggestions are welcome

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...