I have 4 VMs - RHEL 5.6, Windows 7 64-bit Enterprise, Windows Server 2008 R2 64-bit, and Windows XP 32-bit w/SP3.
I learned that setting up a Splunktcp receive port enabled the Windows systems to successfully send fully readable tcp packets on Splunk server (running on the 2008 server).
For some reason, though, the same configuration does not seem to be working on my RHEL VM. tcpdump shows activity leaving the RHEL box, but the Summary window of Splunk server does not show any updates for syslog. The most recent updates are only for the Windows systems.
How do I fix this?
I found another posting here that suggested I copy the $SPLUNKHOME/apps folder and have it overwrite the SUF $SPLUNKHOME/apps folder, then restart splunk on the RHEL box. I tried this, but no difference.
Thanks for any help here.
Correct me if I'm wrong, but you're trying to send logs from the RHEL 5.6 client via a universal forwarder to your splunk server (indexer). You see data going out from the client, but nothing showing up in the server? While I may not have your solution, a good troubleshooting step would be to go to the splunk indexer and open the splunk.log file under $SPLUNK_HOME/var/log/splunk and see if you're getting any major errors.
If that log is clear, I would check the outputs.conf($splunk_home/etc/system/local/) file on the client and ensure it's pointing to the correct server and port: "server = splunk:9997" where splunk is the servername (should be able to ping it) and 9997 is the receive port you specified under Receive data on the server. More information on configuring the outputs.conf can be found at http://docs.splunk.com/Documentation/Splunk/latest/admin/Outputsconf
I did all of this, and nothing unusual was seen. I also reviewed one of the server logs, and compared, word-for-word, the splunk entries between the linux system and Windows. Everything was identical. No errors anywhere.
I finally elected to test splunk directly on the Linux server, removed Splunk server from the Win 2008 R2 server, and had all forwarders pointing to the RHEL 5.6 server, where splunk server was now installed and running.
I configured the receiver to listen on port 9997 (splunktcp receiver) and in almost no time, the Windows systems appeared in Splunk Search. Over a 30 minute period, not once did I see anything in the logs from the Linux server itself.I reviewed the Management configuration options, and I just couldn't find anything that would prevent the indexer/database from showing/accepting Linux logs.
I reviewed tcpdump and the splunk logs. Absolutely nothing out of the ordinary from what I could see.
I have the most recent (within the last couple of months) of Server and the Universal Forwarder.
Is it possible you're sending the logs to a non default index? try specifying "index=* host=<