topic Re: Splunk, VMWare, Syslog and non-default port in Getting Data In

Splunk, VMWare, Syslog and non-default port

mfrost8 — Wed, 10 Nov 2010 06:07:44 GMT

We're trying to setup some test monitoring of a VMWare ESX host (not ESXi). Because our Splunk instance does not run as root, I setup the UDP listener port to be something above 1024.

However, I'm not able to find anything about syslog configuration on an ESX server that shows how one might configure it to send to a remote syslog host on any port other than 514. So I don't know if it just won't work or not (the VMWare is going to try something I suggested in that regard tomorrow), but I'm not all that hopeful.

So if I have to enable a UDP listener in Splunk on port 514, I assume that means I would now have to find a way to run Splunk as root rather than the non-privileged user I'm doing this as now?

This becomes an issue because my team, who administers Splunk, are not the corporate sysadmins and as such are not given root privileges.

Re: Splunk, VMWare, Syslog and non-default port

Lowell — Wed, 10 Nov 2010 07:22:09 GMT

Would running a service such as syslog-ng be more acceptable to your admins? There's no reason why you can't have a standard syslog daemon listen for inbound syslog events and store them in files which splunk is configured to monitor. This is in fact a configuration that I've seen recommended by splunk a number of places.

I suspect there could be a way to do this with some kind of local firewall trick. You may be able to setup iptables (or whatever is appropriate for your system), to take traffic destined to UDP port 514 and redirect that to a local UDP port above 1024. I'm not real fluent with that kind of thing, but it seems like something like this should be possible.

Re: Splunk, VMWare, Syslog and non-default port

dwaddle — Wed, 10 Nov 2010 07:41:05 GMT

The firewall trick Lowell mentions is documented in another answer here: http://answers.splunk.com/questions/6118/how-can-i-receive-syslog-udp-514-events-with-a-non-root-splunkd.

Re: Splunk, VMWare, Syslog and non-default port

mfrost8 — Wed, 10 Nov 2010 23:17:00 GMT

The servers in question (Linux) all have syslog-ng. I like both approaches, but I suspect that asking the syadmin team to custom-configure syslog to send to us might meet with some resistance despite the fact that they don't really look at it much. Also, they have a tendency to overwrite our local configs without noticing. (fschange monitor!) I do like the sound of the firewall change though and will check that out.

Thanks Lowell and dwaddle!