topic Re: Universal Forwarder in Getting Data In

Universal Forwarder

butlerm494 — Wed, 02 May 2018 13:22:26 GMT

Before I start this is a serious case of blind leading the blind.

Currently we have a VMware running Windows Server 2016 hosting Splunk Enterprise, to date we have managed to get the forwarder installed on Windows 7, Windows 2003, Windows 2008, Solaris and Mint Linux (Just for a laugh). Without much administration it all works well, but we come to RHEL 7, for some reason we can not get it work, everything appears to be okay. Installed the RPM forwarder, but nothing appears to be happening.

As this is a test system we have disabled both Server and Client firewalls, can ping the server in both directions - but we can't seem to get it to work. The only thing that we have managed to find using "google" is a potential issue with SELINUX so we have disabled that.

Any suggestions as this would save the sanity of the "intern"

Re: Universal Forwarder

skoelpin — Wed, 02 May 2018 13:35:13 GMT

Can you post your inputs.conf stanza? Also, what do the forwarder log files say?

You can go to /opt/splunkforwarder/var/log/splunk/splunkd.log and take a look

Re: Universal Forwarder

p_gurav — Wed, 02 May 2018 13:38:13 GMT

Is there any error in _internal logs in /opt/splunkforwarder/var/log/splunk/splunkd.log?

Re: Universal Forwarder

butlerm494 — Wed, 02 May 2018 13:45:31 GMT

I can't post the logs since they are on a standalone system, all we have on the universal forwarder inputs.conf is
[default]
host = localhost.localdomain
Any suggestions are welcome

Re: Universal Forwarder

xpac — Wed, 02 May 2018 13:50:33 GMT

Try this on the command line:

/opt/splunkforwarder/bin/splunk list forward-server

It should show you if the UF has successfully connected to any configured destination server.

Also, do you get ANY logs from the forwarder at all, if only _internal logs?

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

Re: Universal Forwarder

butlerm494 — Wed, 02 May 2018 13:55:59 GMT

Thank you for your comment,
I tried that command and it lists the ip address of the machine on which splunk enterprise is installed under active forwards.
Any other suggestion is appriciated

Re: Universal Forwarder

butlerm494 — Wed, 02 May 2018 13:58:52 GMT

Thank you for your answer.
All I have in the inputs.conf file is
[default]
host = localhost.localdomain.
I can't post the log file since the system is on a standalone machine.
Any suggestion is welcome

Re: Universal Forwarder

xpac — Wed, 02 May 2018 13:59:06 GMT

On your search head/indexer, see if you get ANY data of that forwarder (you should, if the indexer shows up as "active forwards"). Try it like this:

| tstats prestats=t count where (index=* OR index=_*) AND host=yourforwardername by _time index
| timechart count by index

Re: Universal Forwarder

xpac — Wed, 02 May 2018 14:00:48 GMT

If you don't have anything else in your inputs.conf, you simply didn't setup any inputs. There is no data going to come because you didn't tell it what to collect. 😉

Re: Universal Forwarder

butlerm494 — Wed, 02 May 2018 14:03:33 GMT

but splunk enterprise should still be able to see the forwarder right? Instead I have no clients phoning home.

Re: Universal Forwarder

xpac — Wed, 02 May 2018 14:06:36 GMT

No, not by default. You need to configure your UF with the IP of the deployment server, they don't call home by default.
You could do this by doing /opt/splunkforwarder/bin/splunk set deploy-poll YOURSERVER:8089 (and maybe restarting).
You could also create a seperate app (this is the clean way!) with a deploymentclient.conf like this:

```
[deployment-client]

[target-broker:deploymentServer]
targetUri= YOURSERVER:8089
```

YOURSERVER has to be replaced with the IP or DNS name of your Splunk instance.

Re: Universal Forwarder

butlerm494 — Wed, 02 May 2018 14:13:45 GMT

I have also already set the deploy-poll and restarted afterwards. The forwarder still does not appear on splunk enterprise

Re: Universal Forwarder

xpac — Wed, 02 May 2018 14:23:26 GMT

Can you use tcpdump on the Splunk Enterprise instance to check if you get any communication from that instance to TCP port 9997?
Also, did you try to check for any logs with my tstats command posted in the other comment?

Re: Universal Forwarder

butlerm494 — Wed, 02 May 2018 14:27:00 GMT

I have TcpOutputFd - read error. Connection reset by peer
tcpoutputProc - connection to closed. Read error. Connection reset by peer.

Re: Universal Forwarder

skoelpin — Wed, 02 May 2018 16:19:28 GMT

Once again, you need to see what the forwarder logs are saying to troubleshoot your issue.. You claimed to install the UF on a RHEL server, so you can either look on that RHEL server under the path I gave you above, or if you are forwarder your UF log files, you can look in Splunk. We are unable to help you until you look