Getting Data In

Can I have a universal forwarder listen on port UDP 514, if the indexer is already doing that?

lucamarc
Path Finder

Hello,

I think the subject summarizes. I searched for answers, but could not find a clear one.
I have my only indexer already configured to listen to UDP 514.

Unfortunately, now I have a second class of devices for which I cannot change the syslog port. I installed a Universal forwarder on a remote server, to collect the logs from these devices and forward them to the indexer.

I'm probably missing something really trivial, but when I go to add the new server class as a forwarded data input, at the very end port UDP 514 is not accepted with error :

Parameter name: UDP port 514 is not available.

But, I'm not trying to collect data on UDP 514 directly from the indexer, in which case that error would make sense... I'm forwarding data from a different machine, which I understand communicate with the indexer on an entirely different port (8089 if I'm not mistaken.

Isn't one of the major point of having forwarders the ability to re-use the same port? Like I said, I must be missing something. 🙂

Thanks,
Luca

0 Karma

JeremyHagan
Communicator

Another option would be to send the data in on 514 and then use some other criteria to override the sourcetype of the data. It is a normal problem for many different types of devices to send syslog on port 514 which you then want to separate into different sourcetypes. Here is how you do it:

  1. Set up a generic UDP listener for 514 into a generic syslog source type (make sure your reverse DNS is working)

    [udp://514]
    connection_host = dns
    sourcetype = syslog
    index = myindex

  2. Set up a sourcetype override in transforms.conf similar to the following

    [new_sourcetype]
    SOURCE_KEY = MetaData:Host
    DEST_KEY = MetaData:Sourcetype
    REGEX = (?i)(myhostname)
    FORMAT = sourcetype::new_syslog

  3. In your props.conf link the syslog sourcetype with the tranform
    [syslog]
    TRANSFORMS-sourcetype_override = new_sourcetype
    LOOKUP-rdns = dnslookup clientip AS host OUTPUT clienthost AS dnsHostname

aaraneta_splunk
Splunk Employee
Splunk Employee

@lucamarc - Did one of the answers below or comments above help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If it was a comment, please mark which one it was so that I can convert it to an answer to be accepted. If you still need help, please leave a comment with more feedback. Thanks.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Parameter name: UDP port 514 is not available.

Since you're on windows and experiencing this, most likely the issue is that another program / service is bound to the port.

In command prompt with Splunk stopped, you can run this command:

netstat -an

If you see LISTENING on *:514 or 0.0.0.0:514 etc then some other program or service is already using the port. Most probably some form of an SNMP service...

0 Karma

jkat54
SplunkTrust
SplunkTrust

I highly recommend against running Splunk as root.

Instead you can use ip tables to forward incoming port
514 traffic to
A higher port like 5514 and then configure Splunk to listen on 5514. This way you have to make your iptables edit as root but you can still run Splunk as a less priviledged user.

iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-port 5514

This will redirect traffic on all interfaces so if you’re doing some other stuff with the local loopback that might break. You can also specify an interface in the rule as follows:

iptables -t nat -A PREROUTING -i eth0 -p udp --dport 514 -j REDIRECT --to-port 5514

This will only do the redirect for traffic on the eth0 interface.

jkat54
SplunkTrust
SplunkTrust

@lucamarc the inputs.conf (not input.conf) with the udp input goes on the forwarder. The forwarder also needs an outputs.conf that specifies the indexers and their receiving ports. Then on the indexers you will have an inputs.conf that specifies port 9997 ([splunktcp://9997])

0 Karma

jethompson_splu
Splunk Employee
Splunk Employee

Hello lucamarc,

As I understand your question you are wanting to obtain Syslog's from Devices using Port 514 using UDP with your Splunk Installation. If you are wanting to get the Syslogs from devices that you are not able to install the UF (Universal Forwarder) on then you will need to setup a Receiving Input on one of your Splunk Instances.

The following link will provide documentation on setting up a Receiver:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Enableareceiver

Now there is a caveat to this type of configuration and the following provides more information in regards to that.

• As you are wanting to monitor port 514 which is below port 1024 you would have to run that process as Root.
(On *nix Installations a “Regular User” cannot start a process on ports below 1024. Only Root can start processes on ports 0 – 1023
To be able to have a process running on a Port lower than 1024 you would have to run that process as Root which is not “Best Practice”).

So for your configuration you would most likely want to have a UF that is setup on a Segregated Instance that is running UF as Root. This will allow you to forward the Data from those Devices to the UF, the UF would then process that Data and forward it to the Indexer accordingly.

Jeff Thompson

lucamarc
Path Finder

Hi,
I'm running Windows (should have stated that from the start). Windows Server 2012 R2 for the indexer.
Does that make things any easier?

Luca

0 Karma

jethompson_splu
Splunk Employee
Splunk Employee

Hi Luca,

The version of OS should not matter, the only difference would be the running of the Process (and as you are using Windows the "Root" issue does not effect your setup).

You will want to make sure that there is no Firewall blocking the traffic for port 514 between the devices and your Splunk instance that is to be collecting those logs. You would also want to make sure that Splunk is actually listening on the ports configured.

The fastest testing method would be to use NetCat and Telnet. You would use NetCat on the Instance that you are pushing logs to, and from the device that is sending that Log Data you would use Telnet to connect. You would use something along the following for the Telnet to NetCat connection test for UDP Traffic:

netcat -ul 514

--- the u tells Netcat to use UDP. the l tells Netcat to "Listen" on the specified port number.

Using Telnet to connect you would use something along the following:

telnet "Host" 514

From the information provided the "Host" for the Telnet command would be: 10.0.50.21 -- If you are able to connect to the Listening NetCat process you should be able to type in the Telnet Session and see that Text displayed on the NetCat host(You may not see that text displayed on the Telnet Host.

You should also make sure that your Device that is pushing the logs out is actually pushing data. Now if you are able to see traffic flowing and still not seeing any data being processed you will want to double check the Splunk Logs for possible issues. Those logs should be located in the var/log Directory of your Splunk Installation.

Jeff Thompson

0 Karma

jkat54
SplunkTrust
SplunkTrust

OS type does matter. Consoder the fact that only root can open ports lower than 1024 on Linux servers by default.

0 Karma

jkat54
SplunkTrust
SplunkTrust

what OS is your Universal Forwarder running?

0 Karma

lucamarc
Path Finder

I installed one on an existing Server 2008 R2 VM (10.0.50.21).

I tried editing input.conf on the indexer here:
/etc/deployment-apps/ruckus/local/input.conf
with the line:
[udp://10.0.50.21:514]

and restarting splunk, but events aren't showing up on the indexer.

Thanks,
Luca

0 Karma

horsefez
SplunkTrust
SplunkTrust

Hi,

could it be that you are mistaking a universal forwarder (uf) for a heavy forwarder (hf)?

I think you want to configure a second server that is able to listen on a specific port for syslog udp on 514.
To do that you need to install a heavy forwarder on the remote server. UF's aren't able to do that in my opinion.

Regards,
pyro_wood

0 Karma

lucamarc
Path Finder

Hi pyro_wood,

Thanks for taking the time to answer. 🙂 Most of the docs I looked at were about light or universal forwarders. I got the sense that LFs were being phased out, so I though UFs are it. From what I read so far, HF is basically a full instance of Splunk Enterprise, with forwarding enabled.

Just to make sure I got this right, the moment you have more than one type of devices for which you cannot change the syslog port from default, you need multiple Splunk Enterprise instances? Also begs the question, would that be another Splunk license?

Thanks,
Luca

0 Karma

horsefez
SplunkTrust
SplunkTrust

Hi Luca!

Yes, HFs are basically a full instance of Splunk Enterprise. They come with their own license, you don't need to buy any extra. In some cases you need to point the heavy forwarder towards your license master, if you have configured one.

If you have multiple systems that can only forward to syslog:514 you could do shenanigans with e.g. ip-tables on linux
Like... have multiple udp stanzas for different ports and then redirect syslog:514 port depending on the src-ip to a different internal port.
I had to do it in the past, but I hated it.

Sadly, splunk isn't able to have multiple udp stanzas for the same port and different systems.

lucamarc
Path Finder

Thanks for explaining that. I think I'll setup another VM and install a Heavy Forwarder on it.

Luca

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...