Deployment Architecture

Can Universal Forwarder listen to an UDP port and forward to the indexer?

marcos_eng1
Explorer

Hello Splunkers,

I have my firewall sending its logs to a CentOS server where I have the Splunk Universal forwarder configured to listen to UDP 514 and forward it to the indexer. Although I have reviewed the configuration I wasn't able to find the reason it is not working.

Note: I have tested the inputs and output.conf and It is working for the files I'm monitoring.

What am I missing here?

Any help would very much be appreciated!

Labels (3)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

just check that you have "nmap-ncat-2:7.70-5.el8.x86_64 : Nmap's Netcat replacement" or some other Netcat installed and then this should work.

You could also check if splunk is listening udp 514 port by (as root, w/o you don't see processes)

netstat -napu | egrep splunkd
udp   0   0 0.0.0.0:514       0.0.0.0:*               13866/splunkd

My proposal is that (if possible) change port to something > 1024 and then run splunkd as non root user. And it this is not possible then you must run splunkd as root.

r. Ismo

View solution in original post

ephemeric
Contributor

Using unreliable UDP is a last resort. I would recommend using a SUF (Splunk Universal Forwarder) installed on your syslog host. Add a monitor input on said host to read whatever log files you require (/var/log/*) and connect to TCP 9997 on the indexer.

Add a forward-server to your SUF, that way you use TCP with TLS if required. You can also enable `UseAck` in the SUF `outputs.conf` for audit grade event collection.

If UDP is a must: you said you're running Splunk as root so port 514 UDP should work. Check your splunkd.log for errors. Check metrics.log for data. Make sure no other process is using port 514.

Try:

 

lsof -nP -iUDP:514
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 30071 root 3u IPv4 11494632 0t0 UDP *:514
rsyslogd 30071 root 4u IPv6 11494633 0t0 UDP *:514

ss -tnlp | grep rsyslogd
LISTEN 0 25 *:514 *:* users:(("rsyslogd",pid=30071,fd=5))
LISTEN 0 25 [::]:514 [::]:* users:(("rsyslogd",pid=30071,fd=6))

 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Btw is this normal OnPrem node or e.g. AWS node?
r. Ismo
0 Karma

marcos_eng1
Explorer

On Premisses

0 Karma

marcos_eng1
Explorer

Thanks for advising. My UF, HF and IDX are in the same LAN and my Linux Firewall are disabled. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Then it’s time to use tcpdump and look if there is traffic or not. 
If you are not familiar with it, then here is some examples https://hackertarget.com/tcpdump-examples/

r. Ismo

0 Karma

marcos_eng1
Explorer

I already checked with tcpdump command and there is traffic.

 

tcpdump -i any udp port 514

0 Karma

splunkcol
Contributor

I know it is not logical and it does not make sense but I remember that one of the things I tried before solving was

sudo firewall-cmd --permanent --add-port = 514 / udp
sudo firewall-cmd --permanent --add-port = 514 / tcp
sudo firewall-cmd --reload

(the server firewall was also disabled when I presented this same error)

when I checked the syslog path again they were already loading the logs

if it does not work just inactivate the firewall again

I don't want to waste your time, I just want to help, as you know in this world there are things that make no sense

in my case the sniffer detected the traffic and netstat -an | grep 514 showed ports in listening mode

Tags (2)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

How about selinux? Is it configured and is in use?  If it is then you must enable splunkd to listen that port.

0 Karma

marcos_eng1
Explorer

Also Selinux is not configured

0 Karma

isoutamo
SplunkTrust
SplunkTrust

I just create UF with UDP listener and send events to indexers with above instructions on RHEL 8.1 with UF 8.0.5 without any issues.

One questions are you running splunk as root or some other users? As I earlier said if it's anyone else than root then port must be creater than 1024. Otherwise it cannot bind to that port.

You could test it on localhost with:

echo "foo bar" | nc -u 127.0.0.1 514

or 

echo "foo bar" | nc -u 127.0.0.1 1514

First cannot connect if splunkd is running as splunk (or any other user than root). Second one works independent of user.

r. Ismo

0 Karma

marcos_eng1
Explorer

Thanks, I am not familiar with shell commands and I was not able to run the tests. Could you guide me to test it with Cent OS?

 


[root@uf01 ~]# echo "foo bar" | nc -u 127.0.0.1 514
-bash: nc: command not found
[root@uf01 ~]# echo "foo bar" | nc -u 172.25.21.17 514
-bash: nc: command not found
[root@uf01 ~]# $echo "foo bar" | nc -u 127.0.0.1 514
-bash: nc: command not found
-bash: foo bar: command not found
[root@uf01 ~]# echo "foo bar" | nc -u 127.0.0.1 1514
-bash: nc: command not found

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

just check that you have "nmap-ncat-2:7.70-5.el8.x86_64 : Nmap's Netcat replacement" or some other Netcat installed and then this should work.

You could also check if splunk is listening udp 514 port by (as root, w/o you don't see processes)

netstat -napu | egrep splunkd
udp   0   0 0.0.0.0:514       0.0.0.0:*               13866/splunkd

My proposal is that (if possible) change port to something > 1024 and then run splunkd as non root user. And it this is not possible then you must run splunkd as root.

r. Ismo

View solution in original post

marcos_eng1
Explorer

Hello Soutamo, after many unsucessful tries I changed the udp port to a higher port and fortunatelly is working now.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Have You  gotten this already to work?

One thing which I haven't seen an answer is that are you running this as root or other user? So what gives output ps -fe | egrep splunk ?

r. Ismo

0 Karma

marcos_eng1
Explorer

[root@uf01 ~]# ps -fe | egrep splunk
root 31641 1 14 11:11 ? 00:02:08 splunkd -p 8089 restart
root 31643 31641 0 11:11 ? 00:00:00 [splunkd pid=31641] splunkd -p 8089 restart [process-runner]
root 31872 31269 0 11:26 pts/0 00:00:00 grep -E --color=auto splunk

0 Karma

marcos_eng1
Explorer

Hello Soutamo, 

I was not able to make it work yet, if you could spare 5 min for a webex call, I would really apreciate.

I am running splunk as root, as follow:

[root@uf01 ~]# ps -fe | egrep splunk
root 21115 1 0 Aug30 ? 00:00:15 splunkd -p 8089 start
root 21117 21115 0 Aug30 ? 00:00:00 [splunkd pid=21115] splunkd -p 8089 start [process-runner]
root 21543 21523 0 00:17 pts/0 00:00:00 grep -E --color=auto splunk
[root@uf01 ~]#

Once I am receiving events from a firewall, do I need to install any add-on on universar forwarder or config the props.conf on HF?

I look forward to hearing from you soon.

Marcos

 

0 Karma

marcos_eng1
Explorer

Firewall is disabled in both servers UF, HF and IDX, also all servers are in the same LAN

 

[root@uf01 bin]# systemctl status firewalld
● firewalld.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)

Aug 17 10:24:50 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Aug 17 10:24:53 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Aug 17 10:33:49 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon...
Aug 17 10:33:50 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

What splunk btool inputs list udp —debug Shows?

0 Karma

marcos_eng1
Explorer


[root@uf01 ~]# cd /opt/splunkforwarder/bin/
[root@uf01 bin]# ./splunk btool inputs list udp --debug
/opt/splunkforwarder/etc/system/default/inputs.conf [udp]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/default/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/system/local/inputs.conf host = uf01
/opt/splunkforwarder/etc/system/default/inputs.conf index = default
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf [udp://514]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf disabled = false
/opt/splunkforwarder/etc/system/local/inputs.conf host = uf01
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf index = test
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf sourcetype = syslog

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Your host = uf01 which come from system/local/inputs.conf. Have you looked those events with that hostname also or only those IPs?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!