VMware Hosts logs do not show up in new Splunk server

[rweales]

We have been using Splunk on a Windows server without issue.  It ingested logs from Vmware hosts, networking hardware, firewalls, Windows events, etc.

We created a new Splunk instance on CentOS Stream 9.  It runs as the Splunk user, so it couldn't use the udp data input of 514.  We set it to 10514 and did port forwarding to get around that.  That works for everything except our VMware hosts.  The logging from them will not show up in the new Splunk server.  All the other devices/logs that want to send on udp 514 show up in Splunk.

The value on the VMware hosts that always worked before was:  udp://xxx.xxx.xxx.xxx:514.  We tried the same with 10514 to no avail.  Is there an issue with receiving logs from VMware hosts and having port forwarding send the data to a different port?

[JohnEGones]

We had this issue with some of our devices for syslog data, the work around is to use a syslog server. If you are comfortable with Linux, then standup a server with rsyslog, do the appropriate configs and then put a UF on the host and have it monitor the log folder/files, etc.

[rweales]

Tcpdump shows syslog coming from everything except our hosts.  I have tried udp/514 and tcp 1514.  Neither show up.  Everything else does show up.  When we had this on a Windows server there was no issue, we didn't have to do anything special - it was coming over on udp/514.

What is the recommended method for ingesting syslog?  We are a small shop and have never had issues with this method in the past.

Also, what distro would you recommend?  This is a new install, so it wouldn't be a stretch to rebuild it.

[PickleRick]

If you can't see your packets in tcpdump output it means that something is wrong earlier. From my limited knowledge of VMware products I'd say you reconfigured your syslog outputs on ESXi but didn't adjust firewall rules to allow outgoing traffic to another port.

As for syslog - there are typically two approaches - one is to receive with a syslog daemon and write to files from which you'd read the events with monitor input on a UF (or even on your Splunk server if your setup is small enough but I'd prefer to separate this functionality to another small host). Another one is to use a syslog receiver (properly configured rsyslog/syslog-ng or SC4S) to receive data over the network and send to a HEC input. While in small setups direct receiving on Splunk server might be "good enough", you lose network-level metadata and it's more frustrating to manage inputs for different types of sources. (not to mention the low port issue).

About the distro... well, that's a bit of a religious issue but depending on your willingness to spend money and other personal preferences I'd consider for production use (in no particular order): RHEL, Rocky/Alma, debian, Ubuntu LTS, SLES, OpenSUSE. No rolling-release distros for prod.

[rweales]

Thanks for the advice.  We are going to go to Ubuntu LTS, and create a separate syslog host then forward to the Splunk server.

[JohnEGones]

Please make sure you're accepting a resolution as an answer.

[PickleRick]

Firstly, check with tcpdump that your events do reach your destination host.

Also - it's not recommended to use a network input directly on splunk component. There are other options of ingesting syslog data.

And I would _not_ use a rolling release type distro like Centos Stream for prod use. But that's just me,

[rweales]

We have setup RHEL 8.10 to be our new Splunk instance. 

As before on CentOS Stream, we get syslog data from everything except the VMware host syslog data...

We still have the Windows Splunk server around, and if we change the Syslog.global.logHost key in the Advanced System Settings on each host back to the Windows Splunk server, then the syslog data from the hosts shows up.

It appears that if splunkd is running under the splunk user, then a port forwarding solution would be required to forward to a higher port for syslog.  However, splunkd is running as root, not the splunk user.

Years ago, we ran Splunk on CentOS 7 and never had this issue.

Is the port forwarding solution the answer here?

[PickleRick]

Again - Firstly, check with tcpdump that your events do reach your destination host. If you don't see the data on the wire no magic within the OS will make it appear out of thin air.

[rweales · Accepted solution]

Right you are, it was a misconfigured fw on the hosts.