Getting Data In

how to listen to port UDP 514 when splunk is not root

yannK
Splunk Employee
Splunk Employee

On linux systems, only a process running as root can listen to ports < 1024.
I want splunk to listen to syslog on UDP 514, while running as a non root user.

1 Solution

yannK
Splunk Employee
Splunk Employee

Run splunk as non root user :
http://docs.splunk.com/Documentation/Splunk/5.0/installation/RunSplunkasadifferentornon-rootuser

Solution 0 : run splunk as root

Solution 1 : redirect the port
Use iptables to redirect the port 514 to another range, like 10514.
The iptable rules are stored in /etc/sysconfig/iptables. It contains the rules which will be loaded during bootup.
The reason why the rules disappeared after reboot is that, they were not saved to this file.
You need to add the rules using the following commands:

iptables -t nat -A PREROUTING -p UDP -m udp --dport 514 -j REDIRECT --to-ports 10514
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 10514
iptables-save

and then, pipe the output from iptables-save into this file.

iptables-save > /etc/sysconfig/iptables

Alternatively, you could also copy the iptables file from/etc/sysconfig from any other working SSIM which already
has the rules for syslog redirect.

And setup splunk to listen to this new port
my favorite config for UDP is

[udp://10514]
sourcetype=syslog
connection_host=ip
  # do not resolve the hostname, it is usually included in the syslog events
queueSize=1MB
  # to add some buffer in case of indexer slow
persistentQueueSize = 5MB
  # to add some disk buffer too 

Solution 2 : use a syslog server
- setup a rsyslog/ syslog-ng server to run as system process, and listen to 514
- have the logs being written to disk, by example a folder per host, with log rotation.
- have splunk monitor those folders.

the advantage if this method is that UDP is non resilient and volatile, so the syslog server will act as a file buffer, and if Splunk is restarting it will catch up once up.

View solution in original post

mcronkrite
Splunk Employee
Splunk Employee

Another example of Solution 1 is doing this redirect from 514 to 5514 :

Poke hole in iptables to allow ports.

Redirect for port 515 up to 5514 which we are listening on.

iptables -I INPUT -p tcp --dport 5514 -j ACCEPT

iptables -t nat -A PREROUTING -d MY.IP -p tcp -m tcp --dport 514 -j DNAT --to-destination MY.IP:5514

iptables -t nat -A PREROUTING -d MY.IP -p udp -m udp --dport 514 -j DNAT --to-destination MY.IP:5514

Be sure to "service iptables save" after modifying iptables, or modify etc/sysconfig/iptables directly.

0 Karma

dwaddle
SplunkTrust
SplunkTrust
0 Karma

yannK
Splunk Employee
Splunk Employee

Run splunk as non root user :
http://docs.splunk.com/Documentation/Splunk/5.0/installation/RunSplunkasadifferentornon-rootuser

Solution 0 : run splunk as root

Solution 1 : redirect the port
Use iptables to redirect the port 514 to another range, like 10514.
The iptable rules are stored in /etc/sysconfig/iptables. It contains the rules which will be loaded during bootup.
The reason why the rules disappeared after reboot is that, they were not saved to this file.
You need to add the rules using the following commands:

iptables -t nat -A PREROUTING -p UDP -m udp --dport 514 -j REDIRECT --to-ports 10514
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 10514
iptables-save

and then, pipe the output from iptables-save into this file.

iptables-save > /etc/sysconfig/iptables

Alternatively, you could also copy the iptables file from/etc/sysconfig from any other working SSIM which already
has the rules for syslog redirect.

And setup splunk to listen to this new port
my favorite config for UDP is

[udp://10514]
sourcetype=syslog
connection_host=ip
  # do not resolve the hostname, it is usually included in the syslog events
queueSize=1MB
  # to add some buffer in case of indexer slow
persistentQueueSize = 5MB
  # to add some disk buffer too 

Solution 2 : use a syslog server
- setup a rsyslog/ syslog-ng server to run as system process, and listen to 514
- have the logs being written to disk, by example a folder per host, with log rotation.
- have splunk monitor those folders.

the advantage if this method is that UDP is non resilient and volatile, so the syslog server will act as a file buffer, and if Splunk is restarting it will catch up once up.

View solution in original post

koshyk
Super Champion

For ubuntu , save iptables persistently using

sudo apt-get install iptables-persistent
sudo /etc/init.d/iptables-persistent save # For Saving new rules
sudo /etc/init.d/iptables-persistent reload # For reloading

check setup is good by
sudo iptables -L -vt nat

0 Karma

yannK
Splunk Employee
Splunk Employee

PS : another solution is to use a proxy (like nginx) to redirect ports to splunk (with splunk using ports > 1024)

It's the same method for udp/tcp, or even for splunkweb port.

0 Karma

Lowell
Super Champion

For whatever it's worth, in 6.1.1 the same limitation still exists. I was hoping that because 6.1 now starts as root and switches to a named (non-privileged) user during startup that this limitation may have been lifted, but testing shows otherwise.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.