topic Re: Can Universal Forwarder listen to an udp port and forward to Indexer? in Deployment Architecture

Can Universal Forwarder listen to an UDP port and forward to the indexer?

marcos_eng1 — Mon, 31 Aug 2020 19:25:26 GMT

Hello Splunkers,

I have my firewall sending its logs to a CentOS server where I have the Splunk Universal forwarder configured to listen to UDP 514 and forward it to the indexer. Although I have reviewed the configuration I wasn't able to find the reason it is not working.

Note: I have tested the inputs and output.conf and It is working for the files I'm monitoring.

What am I missing here?

Any help would very much be appreciated!

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 04:59:49 GMT

it’s possible and normal situation. If you want to listen port 514 then you must run splunkd as root. Better option is use eg. 1514 and run it as normal user.
inputs.conf is something like

https://community.splunk.com/t5/Getting-Data-In/How-to-configure-different-sourcetypes-for-udp-port-514/td-p/165772
r. Ismo

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 13:31:31 GMT

Thanks for the reply. I still was not able make it work. Can anyone help with the step by step configuration?

Note: I know how to config in full Splunk Enterprise Installation but to collect it with Universal Forwarder, it is my first time.

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 14:29:01 GMT

Can you post your input.conf so we could see and help you?

Basically what you need to do (w/o deployment server).

On Indexer:

- Add receiving port

splunk enable listen 9997

Then on UF:

- add output(s) if needed you could add several servers or just edit those conf-files and then restart splunk on UF.

splunk add forward-server <your indexer IP>:9997

- check that it's working and sending internal logs

splunk list forward-server

- Add input for UDP (easiest to edit wanted inputs.conf, best to create your own app for that)

[udp://:1514] connection_host = dns index = <your index> sourcetype = <your sourcetype> <other params which you want to add>

Then do a restart for that UF's splunkd and it should works.

https://docs.splunk.com/Documentation/Splunk/7.3.3/Admin/Inputsconf

r. Ismo

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 14:20:02 GMT

Sure, here it is the Universal Forwarder inputs.conf:

[udp://514]

connection_host = ip

index = test

sourcetype = syslog

disabled = false

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 14:30:33 GMT

please try

[udp://:514] if you are running splunkd as root otherwise

[udp://:1514] if non root user (cannot bind service to port below 1024!

r. Ismo

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

splunkcol — Tue, 25 Aug 2020 15:41:42 GMT

Hello,

Something similar was presented to me.

Heavy forwarder via syslog

Firewall type source sent logs to the heavy forwarder but the folders that they should have were not created when the logs arrived.

The configuration was fine, TCP and UDP ports 514 were active, but the logs were not arriving.

One of the discards was to use the snifer tcpdump host x.x.x.x where in my case the snifer confirmed reception of traffic

Finally it was the firewall of the heavy forwarder that was blocking somehow

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 16:30:05 GMT

I have changed the stanza to [udp://:514] and still did not work.

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 16:31:50 GMT

Thanks for advising. My UF, HF and IDX are in the same LAN and my Linux Firewall are disabled.

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 16:42:37 GMT

Then it’s time to use tcpdump and look if there is traffic or not.
If you are not familiar with it, then here is some examples https://hackertarget.com/tcpdump-examples/

r. Ismo

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 16:44:45 GMT

I already checked with tcpdump command and there is traffic.

tcpdump -i any udp port 514

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 17:04:42 GMT

What splunk btool inputs list udp —debug Shows?

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

splunkcol — Tue, 25 Aug 2020 17:24:40 GMT

I know it is not logical and it does not make sense but I remember that one of the things I tried before solving was

sudo firewall-cmd --permanent --add-port = 514 / udp
sudo firewall-cmd --permanent --add-port = 514 / tcp
sudo firewall-cmd --reload

(the server firewall was also disabled when I presented this same error)

when I checked the syslog path again they were already loading the logs

if it does not work just inactivate the firewall again

I don't want to waste your time, I just want to help, as you know in this world there are things that make no sense

in my case the sniffer detected the traffic and netstat -an | grep 514 showed ports in listening mode

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 18:23:14 GMT

[root@uf01 ~]# cd /opt/splunkforwarder/bin/
[root@uf01 bin]# ./splunk btool inputs list udp --debug
/opt/splunkforwarder/etc/system/default/inputs.conf [udp]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/default/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/system/local/inputs.conf host = uf01
/opt/splunkforwarder/etc/system/default/inputs.conf index = default
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf [udp://514]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf disabled = false
/opt/splunkforwarder/etc/system/local/inputs.conf host = uf01
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf index = test
/opt/splunkforwarder/etc/apps/_server_app_teste_firewall/local/inputs.conf sourcetype = syslog

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 18:30:57 GMT

How about selinux? Is it configured and is in use? If it is then you must enable splunkd to listen that port.

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 18:33:44 GMT

Firewall is disabled in both servers UF, HF and IDX, also all servers are in the same LAN

[root@uf01 bin]# systemctl status firewalld
● firewalld.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)

Aug 17 10:24:50 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Aug 17 10:24:53 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Aug 17 10:33:49 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon...
Aug 17 10:33:50 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon.

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 18:35:35 GMT

Also Selinux is not configured

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 18:37:46 GMT

Your host = uf01 which come from system/local/inputs.conf. Have you looked those events with that hostname also or only those IPs?

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

marcos_eng1 — Tue, 25 Aug 2020 18:39:59 GMT

Yes. I have looked for anything arriving in my test index and nothing has showed up.

Re: Can Universal Forwarder listen to an udp port and forward to Indexer?

isoutamo — Tue, 25 Aug 2020 21:32:36 GMT

I just create UF with UDP listener and send events to indexers with above instructions on RHEL 8.1 with UF 8.0.5 without any issues.

One questions are you running splunk as root or some other users? As I earlier said if it's anyone else than root then port must be creater than 1024. Otherwise it cannot bind to that port.

You could test it on localhost with:

echo "foo bar" | nc -u 127.0.0.1 514

echo "foo bar" | nc -u 127.0.0.1 1514

First cannot connect if splunkd is running as splunk (or any other user than root). Second one works independent of user.

r. Ismo