First off, I've been using Splunk for about two weeks so I am not all that familiar with how things are suppose to work.
I have Splunk Light installed on Linux (and so far, it is receiving forwarded logs from two windows servers and two linux servers as well as syslogs from our two firewalls. So far, so good.
I am trying to get Splunk to receive logs from our three ESXi hosts (two are 5.5 and one is 6.0) without any luck. I tried forwarding them to the default 514 port on Splunk, but Splunk said it was unable to create a listener on that port (due to not running as root from what I've read), so I configured ESXi to send its syslog to the same listener (on port 33514) that I am using for the firewall syslogs, but it doesn't look like I can modify the firewall on ESXi to send to that port (only 514 and 1514 are options).
So I created a second listener on port 1514 on Splunk, which worked, but when I configured ESXi to send to that host:port combination, Splunk doesn't receive the data.
I read around, and found that there is a Splunk Add-on for VMware - but couldn't find it in my Add-on lists. I'm confused about how to install this, or where.
I also found an OVA for a Splunk DCN and installed that on one of the ESXi hosts. Went through the configuration and got this error:
License master configuration: Fail
In handler 'localslave' : editTracker failed, reason='WARN: path=/masterlm/usage: This license does not support being a remote master. from ip=10.25.1.24'
So now I am just really lost on which method is the recommended method to get ESXi syslogs to a Splunk Light server.
Looking at that diagram, I have Splunk acting as a "Syslog Server" already. Can I not just set ESXi to send its logs to that same server?
You should be able to -- although I have not used "Splunk Light" before.
Have you gone into Settings >> Data Inputs to see if you have added and/or enabled TCP/1514 and UDP/514 for syslog reception?
Other than that, I would try to see if you do receive data on those ports on the Splunk host from your ESXi host with something like "tcpdump".
Also, Splunk's logs in /opt/splunk/var/log/splunk/splunkd.log may also help to troubleshoot.
One more possibility is that syslog may not have the correct time stamps so you may also try to search with "All time" window on Splunk, if you are sure that data should be received in Splunk.
I did try to install the TAs into my Splunk server, but got as far as extracting the gzip into /opt/splunk/etc/apps/ and not knowing what to do from there.
I went into my Splunk Data Inputs to try and add a TCP input on 1514 since I was originally planning on using UDP.
I see that two inputs are already created (possibly by the OVA?):
1514 with a source type of vmw-syslog
1517 with a source type of vclog
Both are disabled, and when I enable the 1514 input, I receive the following error:
Error occurred attempting to enable 1514: In handler 'raw': Could not find writer for: /nobody/SplunkTAesxilogs/inputs/tcp://1514  [/opt/splunk/etc].
I have been using rsyslog as syslog server, so I don't know exactly how the SplunkTAesxilogs add-on is configured -- Try to follow this and see if you could get it going.
My guess is that, SplunkTAesxilogs takes data in. It then will need to either write the data to a file or an internal pipeline to another Splunk module.
Your Splunk Light server may be configured out-of-the-box to forward data to itself for indexing. You may run "splunk btool outputs list" and search for "defaultGroup" to see where your data goes.