Before I start this is a serious case of blind leading the blind.
Currently we have a VMware running Windows Server 2016 hosting Splunk Enterprise, to date we have managed to get the forwarder installed on Windows 7, Windows 2003, Windows 2008, Solaris and Mint Linux (Just for a laugh). Without much administration it all works well, but we come to RHEL 7, for some reason we can not get it work, everything appears to be okay. Installed the RPM forwarder, but nothing appears to be happening.
As this is a test system we have disabled both Server and Client firewalls, can ping the server in both directions - but we can't seem to get it to work. The only thing that we have managed to find using "google" is a potential issue with SELINUX so we have disabled that.
Any suggestions as this would save the sanity of the "intern"
Try this on the command line:
/opt/splunkforwarder/bin/splunk list forward-server
It should show you if the UF has successfully connected to any configured destination server.
Also, do you get ANY logs from the forwarder at all, if only _internal logs?
Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂
Thank you for your comment,
I tried that command and it lists the ip address of the machine on which splunk enterprise is installed under active forwards.
Any other suggestion is appriciated
On your search head/indexer, see if you get ANY data of that forwarder (you should, if the indexer shows up as "active forwards"). Try it like this:
| tstats prestats=t count where (index=* OR index=_*) AND host=yourforwardername by _time index
| timechart count by index
Is there any error in _internal logs in /opt/splunkforwarder/var/log/splunk/splunkd.log?
I have TcpOutputFd - read error. Connection reset by peer
tcpoutputProc - connection to closed. Read error. Connection reset by peer.
Can you post your inputs.conf
stanza? Also, what do the forwarder log files say?
You can go to /opt/splunkforwarder/var/log/splunk/splunkd.log
and take a look
Thank you for your answer.
All I have in the inputs.conf file is
[default]
host = localhost.localdomain.
I can't post the log file since the system is on a standalone machine.
Any suggestion is welcome
If you don't have anything else in your inputs.conf, you simply didn't setup any inputs. There is no data going to come because you didn't tell it what to collect. 😉
but splunk enterprise should still be able to see the forwarder right? Instead I have no clients phoning home.
No, not by default. You need to configure your UF with the IP of the deployment server, they don't call home by default.
You could do this by doing /opt/splunkforwarder/bin/splunk set deploy-poll YOURSERVER:8089
(and maybe restarting).
You could also create a seperate app (this is the clean way!) with a deploymentclient.conf like this:
```
[deployment-client]
[target-broker:deploymentServer]
targetUri= YOURSERVER:8089
```
YOURSERVER has to be replaced with the IP or DNS name of your Splunk instance.
I have also already set the deploy-poll and restarted afterwards. The forwarder still does not appear on splunk enterprise
Once again, you need to see what the forwarder logs are saying to troubleshoot your issue.. You claimed to install the UF on a RHEL server, so you can either look on that RHEL server under the path I gave you above, or if you are forwarder your UF log files, you can look in Splunk. We are unable to help you until you look
Can you use tcpdump on the Splunk Enterprise instance to check if you get any communication from that instance to TCP port 9997?
Also, did you try to check for any logs with my tstats command posted in the other comment?
I can't post the logs since they are on a standalone system, all we have on the universal forwarder inputs.conf is
[default]
host = localhost.localdomain
Any suggestions are welcome