I think the subject summarizes. I searched for answers, but could not find a clear one.
I have my only indexer already configured to listen to UDP 514.
Unfortunately, now I have a second class of devices for which I cannot change the syslog port. I installed a Universal forwarder on a remote server, to collect the logs from these devices and forward them to the indexer.
I'm probably missing something really trivial, but when I go to add the new server class as a forwarded data input, at the very end port UDP 514 is not accepted with error :
Parameter name: UDP port 514 is not available.
But, I'm not trying to collect data on UDP 514 directly from the indexer, in which case that error would make sense... I'm forwarding data from a different machine, which I understand communicate with the indexer on an entirely different port (8089 if I'm not mistaken.
Isn't one of the major point of having forwarders the ability to re-use the same port? Like I said, I must be missing something. 🙂
could it be that you are mistaking a universal forwarder (uf) for a heavy forwarder (hf)?
I think you want to configure a second server that is able to listen on a specific port for syslog udp on 514.
To do that you need to install a heavy forwarder on the remote server. UF's aren't able to do that in my opinion.
Thanks for taking the time to answer. 🙂 Most of the docs I looked at were about light or universal forwarders. I got the sense that LFs were being phased out, so I though UFs are it. From what I read so far, HF is basically a full instance of Splunk Enterprise, with forwarding enabled.
Just to make sure I got this right, the moment you have more than one type of devices for which you cannot change the syslog port from default, you need multiple Splunk Enterprise instances? Also begs the question, would that be another Splunk license?
Yes, HFs are basically a full instance of Splunk Enterprise. They come with their own license, you don't need to buy any extra. In some cases you need to point the heavy forwarder towards your license master, if you have configured one.
If you have multiple systems that can only forward to syslog:514 you could do shenanigans with e.g. ip-tables on linux
Like... have multiple udp stanzas for different ports and then redirect syslog:514 port depending on the src-ip to a different internal port.
I had to do it in the past, but I hated it.
Sadly, splunk isn't able to have multiple udp stanzas for the same port and different systems.
Thanks for explaining that. I think I'll setup another VM and install a Heavy Forwarder on it.
As I understand your question you are wanting to obtain Syslog's from Devices using Port 514 using UDP with your Splunk Installation. If you are wanting to get the Syslogs from devices that you are not able to install the UF (Universal Forwarder) on then you will need to setup a Receiving Input on one of your Splunk Instances.
The following link will provide documentation on setting up a Receiver:
Now there is a caveat to this type of configuration and the following provides more information in regards to that.
• As you are wanting to monitor port 514 which is below port 1024 you would have to run that process as Root.
(On *nix Installations a “Regular User” cannot start a process on ports below 1024. Only Root can start processes on ports 0 – 1023
To be able to have a process running on a Port lower than 1024 you would have to run that process as Root which is not “Best Practice”).
So for your configuration you would most likely want to have a UF that is setup on a Segregated Instance that is running UF as Root. This will allow you to forward the Data from those Devices to the UF, the UF would then process that Data and forward it to the Indexer accordingly.
I'm running Windows (should have stated that from the start). Windows Server 2012 R2 for the indexer.
Does that make things any easier?
I installed one on an existing Server 2008 R2 VM (10.0.50.21).
I tried editing input.conf on the indexer here:
with the line:
and restarting splunk, but events aren't showing up on the indexer.
The version of OS should not matter, the only difference would be the running of the Process (and as you are using Windows the "Root" issue does not effect your setup).
You will want to make sure that there is no Firewall blocking the traffic for port 514 between the devices and your Splunk instance that is to be collecting those logs. You would also want to make sure that Splunk is actually listening on the ports configured.
The fastest testing method would be to use NetCat and Telnet. You would use NetCat on the Instance that you are pushing logs to, and from the device that is sending that Log Data you would use Telnet to connect. You would use something along the following for the Telnet to NetCat connection test for UDP Traffic:
netcat -ul 514
--- the u tells Netcat to use UDP. the l tells Netcat to "Listen" on the specified port number.
Using Telnet to connect you would use something along the following:
telnet "Host" 514
From the information provided the "Host" for the Telnet command would be: 10.0.50.21 -- If you are able to connect to the Listening NetCat process you should be able to type in the Telnet Session and see that Text displayed on the NetCat host(You may not see that text displayed on the Telnet Host.
You should also make sure that your Device that is pushing the logs out is actually pushing data. Now if you are able to see traffic flowing and still not seeing any data being processed you will want to double check the Splunk Logs for possible issues. Those logs should be located in the var/log Directory of your Splunk Installation.