Windows 2016 / Spunk 188.8.131.52
Today I have installed Splunk and configured it as heavy forwarder ref. https://docs.splunk.com/Documentation/Splunk/8.0.4/AddMcafeeCloud/InstallHWF Currently I'm able to search the _internal index and see the splunkd.log events from the host, so forwarding and recieving should be just fine.
On the heavy forwarder I have defined a TCP port 514 without host limitations. Sourcetype and index is also defined.
[tcp://514] connection_host = dns index = network sourcetype = bluecoat:proxysg:access:syslog
But when searching the index from the searchhead I'm not able to see any syslog events. I do assume that our network administrator has defined the proxy to send syslog to correct servername/port, but just in case I do also use the Kiwi Syslog Message Generator to test sending messages as well but ... nothing. Searching for the message text: nothing, source ip: nothing.
I'm on Windows, so using netstat | findstr 514 I do see that there is a connetction from the server which I use to send the test message from. A bit lost right now....
I do see that you are using Windows (sorry about that), but the best option that I have ever seen for syslog data is the Splunk Connect for Syslog app available from Splunkbase. It would be installed on a Linux server, so if your environment doesn't support Linux, or you have no clue how to use a Linux machine, then you would not want to use it. It doesn't require an HF to send the data to splunk, only an HEC endpoint (whether that is an HF or an indexer (or cluster), so it is light weight and fast (I've had 4 times the throughput of a UF and 20 times the throughput of an HF on the same server using this method and still not maxed out the server). I would seriously look into this method. Syslog straight into a splunk machine is not a good idea, you will have packet loss if you do. Splunk is slow to restart, so any time you try to update or restart that type of HF you will get packet loss during that whole time.
Something to think about.
For the time being I'm stuck on Windows, to bad because the Splunk Connect for Syslog looks pretty amazing.
But - after a couple of hours not looking into the issue, and then giving it a second try I realized that the config I have created will not send the data from the HF to both targets, but only one of them. Commenting out the config in props.conf, outputs.conf and transforms.conf and ... ok, now the HF forwards to the indexer. That was plain and simple. Will create a new question for the follow up topic.